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Executive Summary 


TAACCCT program descnption and activities 


CNA Education served as the third-party evaluator for the Connecticut Health and 
Life Sciences Career Initiative (HL-SCI), a statewide initiative funded from September 
2012 to March 2016 through a Trade Adjustment Assistance Community College and 
Career Training (TAACCCT) grant from the U.S. Department of Labor. The purpose of 
HL-SCI is to prepare workers throughout Connecticut for high-wage, in-demand jobs 
in health and life science fields, with a particular focus on recruiting veterans and 
workers who are unemployed, underemployed, or displaced by foreign trade. The HL- 
SCI Consortium consists of five community colleges (Capital, Gateway, Manchester, 
Middlesex, and Norwalk), the online Charter Oak State College, Eastern Connecticut 
State University, and local workforce investment boards. 


HL-SCI partners developed new health and life science certificate and associate’s 
degree programs and revised previously existing programs. The new and revised 
programs incorporate several core programmatic components, including online and 
hybrid coursework; online booster modules providing supplemental instruction in 
key course concepts; increased prior learning assessment (PLA) to award students 
credit for prior noncredit coursework, training, and knowledge; and enhanced job 
and internship placement services. 


Evaluation design summary 


Implementation study design 


At the beginning of the grant, the Consortium set 17 goals for implementing HL-SCI 
deliverables (such as the number of new certificate and degree programs to be 
created). To measure capacity building, the evaluation team collected data from HL- 
SCI annual reports to the U.S. Department of Labor to track actual performance 
relative to grant goals. 


The implementation evaluation also examined the five key components of HL-SCI, as 
well as the structures and strategies supporting it. The implementation evaluation 
sought to answer two principal research questions: 
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1. What are the challenges to the implementation of each of the HL-SCI grant 
components (booster modules, online and hybrid courses, PLAs, and 
internship/job placement services)? 


2. What are promising practices related to the implementation of each of the HL- 
SCI grant components (booster modules, online and hybrid courses, PLAs, and 
internship/job placement services)? 


This report includes a summary of feedback collected through interviews and focus 
groups throughout the grant period, the spring 2016 results of a student survey 
administered to all HL-SCI participants, and the results of a Survey Monkey 
questionnaire administered to students at the conclusion of each booster module. 


Impact study design 


To evaluate the overall impact of the HL-SCI grant, the impact evaluation sought to 
answer the following research questions: 


1. (College persistence) Do HL-SCI participants persist in their colleges at higher 
rates than students who enrolled in similar programs prior to the 
implementation of the intervention? 


2. (Credential completion) Do HL-SCI participants complete certificate or degree 
programs at a higher rate than students who enrolled in similar programs 
prior to the implementation of the intervention? 


3. (Credit accumulation) Do HL-SCI participants complete more credits than 
students who enrolled in similar programs prior to the implementation of the 
intervention? 


All of these outcomes can be measured reliably using databases of student records 
from each of the Consortium colleges. The impact evaluation also includes separate 
effectiveness analyses for three components of HL-SCI: PLAs, booster modules, and 
online and hybrid courses. 


The analysis employed a quasi-experimental approach to compare the outcomes of 
HL-SCI participants with those of participants in the same or similar programs at 
their colleges prior to the start of the grant. Students in the comparison group were 
carefully selected to match the students in the intervention group, using coarsened 
exact matching. The outcomes for the two groups were compared after one and two 
years of program participation. These impacts should not be interpreted as causal, as 
there may be unobserved differences between the HL-SCI and comparison groups 
that are not accounted for in the analysis and may influence student outcomes. 


The evaluation team also conducted a quantitative descriptive analysis that assessed 
Consortium performance on nine student outcome goals (such as the number of 
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participants completing a TAACCCT-funded program of study) using data from HL- 


SCI annual reports to the U.S. Department of Labor. 


Implementation findings 


The Consortium exceeded all of the final deliverable goals (table $1). This indicates 
that the Consortium greatly expanded access to HL-SCI programs of study, the types 


of programs offered, and the availability of academic supports. 


Table S1. Summary of actual performance relative to HL-SCI grant deliverable goals 


Goal 

Grant deliverable Target Actual met? 
1. Number of new certificatesand degrees 15 20 Yes 
2. Number of revised certificatesand degrees 34 48 Yes 
3. Number of students enrolled in new certificatesand degrees 600 637 Yes 
4. Number of existing programs of study revised so that 30 44 Yes 

credentials are stacked / latticed 
5. Number of students enrolled in revised certificates and 2,700 4,371 Yes 

degrees 
6. Number of students taking online skills assessments 1,350 2,478 Yes 
7. Total math and science booster modules 140 154 Yes 
8. Number of students taking math and science booster modules 3,200 4,792 Yes 
9. Number of new online and hybrid courses offered 60 71 Yes 
10. Number of online modules with feedback/assessment 450 789 Yes 
11. Total number of students taking online and hybrid courses 2,400 3,248 Yes 
12. Number of students receiving PLA credits 675 1,629 Yes 
13. Total number of PLA credits awarded 10,000 15,164 Yes 
14, Number of additional noncredit programs recognized by 36 57 Yes 

CCAP 
15. Number of additional credits available by CCAP 324 719 Yes 
16. Number of participants placed in intemships 360 2,412 Yes 
17. Number of participants receiving job placement services 2,000 4,248 Yes 


The evaluation team also identified strengths and weaknesses of the program and its 
components based on the qualitative data analysis. Key findings include: 


e Program enrollment and recruitment. Most students learned about their 
programs independently, although those who did learn about their programs 
from faculty or staff members at the college found this input to be very 


influential in their decision to enroll. 


e Booster modules. Most students who had taken booster modules found them 
useful because they provided another method through which to learn course 


material. 


e Online and hybrid courses. Most participants preferred in-person courses to 
online and hybrid formats because in-person courses allow for more interaction 
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between students and professors. Some students appreciated online courses 
because they were convenient and allowed students to complete content at their 
own pace. 


e Prior learning assessments (PLA). Most students believed that the PLA process 
was easy to understand, they received the right amount of credit, and they would 
be able to complete their programs more quickly. 


e Employment and placement services. Students liked that clinical experiences 
were hands-on and allowed them to apply what they had learned in the 
classroom. Most students found employment and placement services helpful 
although some students expressed the need for additional career guidance. 


Participant impactsand outcomes 


The Consortium exceeded seven of the nine student outcome goals (table S2). The 
two unmet goals pertained to employment outcomes for program completers who 
were not employed at initial program enrollment. However, many HL-SCI participants 
were still enrolled in their programs of study at the end of the grant period, so there 
was insufficient follow-up time to assess these outcomes for the majority of 
participants. 


Table S2. Summary of actual performance relative to student outcome goals 


Student outcome Target Actual Goal 

met? 
1. Unique participants served 2,700 4530 #£Yes 
2. Number of participants completing a HL-SCI program of study 783 1,069 Yes 
3. Number of new and continuing participants retained in a HL- 2,244 5,152 Yes 

SCI program of study (not unique) 

4. Number of participants completing credit hours 2,430 3,491 Yes 
5. Total credentials eamed 861 1,096 Yes 
6. Number of completers enrolled in further education 196 234 Yes 
7. First-time employment for completers 587 388 No 
8. Retention in first-time employment for completers 470 283 No 
9. Number of participants receiving a wage increase 405 1408 Yes 


Key findings from the impact evaluation include: 


e HL-SCI students and comparison students performed similarly on all outcomes 
(table S3). 


e Results for the intervention group also tended to be similar by program category, 
except that HL-SCI students in science programs completed approximately one to 
two courses fewer than HL-SCI students in all programs after two years of 
program enrollment. 
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e HL-SCI participants who received PLA credit were more likely to complete a 
credential within one or two years than participants without PLA credits. 


e HL-SCI participants who received PLA credits were less likely to persist after the 
first year than participants without PLA credits. 


Table $3. Summary of the overall impact of the HL-SCI grant on the predicted 
probability of tudent outcomes 


Regression-adjusted mean or 
predicted probability 


Years of HL-SCI Comparison 

exposure group group Difference 
College persistence lyear 68.9% 68.8% 0.1% 

2 years 66.2% 65.5% 0.7% 
Credential completion lyear 1.6% 1.1% 0.5% 

2 years 17.1% 15.3% 1.8% 
Credits accumulated lyear 14.74 15.01 -0.27 

2 years 27.46 26.17 1.29 


We also examined course completion and performance outcomes for the booster 
module and online and hybrid course HL-SCI components. HL-SCI participants 
enrolled in these courses were matched to other students enrolled in the same 
course before the HL-SCI component was added. Key findings include: 


e There were no differences in course completion rates, which were greater than 90 
percent before and after the HL-SCI component. 


e There is some evidence that course grades were higher for students in online and 
hybrid courses than for students enrolled in the same course in traditional in- 
person format. 


Conclusions 


Using the findings from the evaluation, the evaluation team generated 
recommendations for staff at Consortium colleges. Many of these recommendations 
are general enough that they may inform other groups involved in similar initiatives. 


Program enrollment and recruitment 
e Increase efforts by staff at Consortium colleges to recruit students. 


e Have college staff members guide students in selecting a program of study. 
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Booster modules 
e Maintain and expand student access to booster modules. 
Online and hybrid courses 


e Improve student engagement and interaction with faculty in online and hybrid 
courses. 


e Continue to offer online courses to ensure flexibility for students. 


e Further investigate why students perceive that they learn more in in-person 
courses. 


Prior learning assessments (PLA) 


e Ensure that students who might benefit from PLA take advantage of the 
opportunity. 


e Ensure that the process for awarding credit is straightforward and transparent. 
Employment and placement services 


e If possible, ensure that clinical hours are flexible so that students can meet other 
obligations. 


e Expand access to college and career guidance for continuing students and 
graduates. 


Implications for future workforce and education 
research 


e Overall, results were similar on all outcomes for the HL-SCI students and the 
matched comparison students. However, this evaluation included only students 
who entered an HL-SCI program of study in the first two years of the grant and 
tracks outcomes for one to two years, so it is possible that it may be too early to 
detect an impact. Future research should examine the impacts of the strategies 
tested under the grant after a longer duration. 


e Results for PLAs were mixed among HL-SCI participants. If implemented 
properly, PLAs should assist students in completing their programs and earning 
credentials more quickly. If, however, PLA credits are granted too freely, students 
may not possess the background they need to master the material in more 
advanced courses. Future research may examine differences in policies for 
awarding PLA credits and the impact on student outcomes. 
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Overview of the HL-SCI Evaluation 


CNA Education served as the third-party evaluator for the Connecticut Health and 
Life Sciences Career Initiative (HL-SCI), which is funded through a Trade Adjustment 
Assistance Community College and Career Training (TAACCCT) grant from the U.S. 
Department of Labor. The purpose of the HL-SCI grant is to prepare veterans; 
trade adjustment assistance (TAA)-eligible workers (that is, those displaced by 
foreign trade); and dislocated, unemployed, and underemployed workers for high- 
paying, in-demand careers in health and life science fields. 


Through the initiative, a consortium of five community colleges,’ the online Charter 
Oak State College, and Eastern Connecticut State University developed new 
certificate and associate’s degree programs, revised existing programs, and put in 
place other strategies to support student success. The new and revised programs 
were developed with input from industry partners to ensure that the programs align 
with labor market demands. Grant funding and program development began in 
October 2012, while program implementation began in spring 2013 and continued 
through the end of March 2016. 


The third-party evaluation used a mixed methods research design to assess the 
implementation and effectiveness of the initiative. The evaluation had three primary 
components: a descriptive component tracking actual performance relative to goals 
outlined in the original grant proposal, a qualitative implementation evaluation, and 
a quantitative impact evaluation. 


First, the descriptive analysis compared actual performance relative to targets for 17 
HL-SCI deliverable goals and 9 student outcome goals using data from HL-SCI annual 
reports to the U.S. Department of Labor. Second, the qualitative component of the 
evaluation examined implementation progress and opportunities for and barriers to 
success within the program. Throughout the grant period, CNA collected feedback 
from students at all five Consortium colleges through interviews, focus groups, and 
surveys. Previous reports also included an analysis of documents provided on 
program implementation throughout the Consortium and case studies of one 


' The five community colleges are Capital, Gateway, Manchester, Middlesex, and Norwalk 
Community Colleges. 
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“flagship” program at each college that was a focus of grant activities (Pearson, 
Daponte, & Cruz, 2015a, 2015b, 2016). Third, the impact evaluation component 
examined the program’s impact on student outcomes, including college persistence, 
credential completion, and credit accumulation. 


Desc nption of the HL-SCI intervention 


HL-SCI provides targeted training leading to industry-recognized certificates and 
associate’s degrees in health and life science fields at five community colleges in 
Connecticut: Capital, Gateway, Manchester, Middlesex, and Norwalk Community 
Colleges. HL-SCI includes a set of five evidence-based program components: 


e New and revised credential programs that align with industry-recognized 
credentials and include stacked / latticed credentials, which help workers 
advance along a career pathway, move up a career ladder (stacked), or shift to 
a related field (latticed). HL-SCI credentials lead to certificates and 
associate’s degrees that can articulate with baccalaureate degree programs. 


e Online booster instructional modules, which provide supplemental 
instruction to help students succeed. Some booster modules review course 
content, while others cover more general topics such as study skills, time 
management, and note-taking. Additionally, some booster modules were 
designed specifically to support veterans; these booster modules cover topics 
such as how to return to campus life and (for college staff) how to advise 
veterans. 


e Online and combined online/traditional classroom (hybrid) courses to 
make credential programs more accessible by giving students greater 
flexibility in scheduling courses. 


e In partnership with Charter Oak State College, the development of a 
standardized prior learning assessment (PLA) system to award students 
with credit for prior credit and noncredit coursework, training, and 
knowledge gained on the job, as well as a coordinated effort to increase 
prospective and current students’ awareness of the PLA system. 


e Internship and job placement services to help students secure internships or 
jobs in their fields of study while enrolled in college and find permanent 
employment following program completion. These services include advice on 
searching for internships and permanent jobs, resume and_ interview 
preparation, and guidance on the soft skills necessary for all health and life 
science occupations. 
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The ultimate goal of TAACCCT grants is to prepare program participants for 
employment in high-wage, high-skill occupations. According to data from EMSI 
Analyst™ presented in HL-SCI’s technical proposal, at the beginning of the grant 
period the state of Connecticut employed approximately 200,000 people in the 
health and life science industries and expected to add 11,000 jobs in these 
industries in the next eight years These data are supported by nationwide job 
growth projections within the healthcare field. An aging population and advances in 
medical technologies continue to drive growth in the healthcare professions. The 
Center on Education and the Workforce at Georgetown University found that 
healthcare had the greatest increase in the number of jobs among all major 
occupation groups from December 2007 to January 2016 (N=2.3 million new jobs). 
Two out of three jobs added in healthcare professional and technical occupations 
during this time required a bachelor’s degree or higher (Carnevale, Jayasundera, & 
Gulish, 2016). 


Data and methods 


The program evaluation used both qualitative and quantitative research methods, 
with a focus on the implementation of specific HL-SCI intervention components and 
the overall effectiveness of the HL-SCI initiative. This section describes the data 
and methodological approaches used to determine the results included in the final 
report. 


Consortium-wide performance relative to goals 


The original HL-SCI TAACCCT grant proposal to the U.S. Department of Labor 
established numerical goals for development of and student participation in each 
component of the HL-SCI grant: the new or revised certificate or degree programs, 
booster modules in math and science, online and hybrid courses, PLA, and 
internships and job placements.’ This report documents actual performance relative 
to the 17 deliverable goals displayed in table 1. The results for these goals are 
included with the implementation evaluation results section of the report. 


* Some of these goals pertain to the total number of students who participated in grant-funded 
activities, regardless of whether they were HL-SCI participants. For example, some students in 
non-HL-SCI programs of study enrolled in courses with booster modules. Other goals pertain 
only to participants of HL-SCI programs of study. The terms “student” and “participant” are 
used to distinguish who is included in each goal. 
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Table 1. HL-SCI grant deliverable goals 


Grant deliverable Target 

1. Numberof new certificatesand degrees 15 
2. Number of revised certificatesand degrees 34 
3. Number of students enrolled in new certificatesand degrees 600 
4. Number of existing programs of study revised so that credentials are 30 

stacked / latticed 

5. Number of students enrolled in revised certificatesand degrees 2,700 
6. Number of students taking online skills assessments 1,350 
7. Total math and science booster modules 140 
8. Number of students taking math and science booster modules 3,200 
9. Number of new online and hybrid courses offered 60 
10. Number of online modules with feedback/assessment 450 
11. Total number of students taking online and hybrid courses 2,400 
12. Number of students receiving PLA credits 675 
13. Total number of PLA credits awarded 10,000 
14, Number of additional noncredit programs recognized by CCAP 36 
15. Number of additional credits available by CCAP 324 
16. Number of participants placed in intemships 360 
17. Number of participants receiving job placement services 2,000 


The grant also outlined nine goals for student outcomes related to enrollment in and 
completion of HL-SCI programs of study and continuing education and employment 
of program completers (table 2).3 The results for these goals are included with the 


impact evaluation results. 


Table 2. HL-SCI grant student outcome goals 


Student outcome Target 
1. Unique participants served 2,700 
2. Number of participants completing an HL-SCI program of study 783 

3. Number of participants retained in an HL-SCI program of study 2,244 
4. Number of participants completing credit hours 2,430 
5. Total credentials eamed 861 

6. Number of completers enrolled in further education 196 

7. Number of completers employed 587 

8. Number of completers retained in employment 470 

9. Number of participants receiving a wage increase 405 


* These student outcome goals include only participants who completed or were retained in an 
HL-SCI program of study. They do not include students who switched to a non-TAACCCT- 
funded program of study. This means that the outcomes reported may underestimate total 


completion and retention rates. 
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Approach to the implementation evaluation 


The implementation evaluation examined the five key components of HL-SCI, as well 
as the structures and strategies supporting it. The implementation evaluation 
sought to answer two principal research questions: 


3. What are the challenges to the implementation of each of the HL-SCI grant 
components (booster modules, online and hybrid courses, PLAs, and 
internship/job placement services)? 


4. What are promising practices related to the implementation of each of the HL- 
SCI grant components (booster modules, online and hybrid courses, PLAs, and 
internship/job placement services)? 


These research questions were addressed using interviews and focus groups with 
students, a student survey administered to all grant participants, and a Survey 
Monkey survey administered to all students who completed math and science 
booster modules. Previous reports also included a survey of Consortium faculty 
members on the development and implementation of booster modules and case 
studies on one program at each college that was a focus of grant activities (Pearson, 
Daponte, & Cruz, 2015b). 


Focus groups and interviews 


The qualitative implementation evaluation included in-depth interviews and focus 
groups conducted either in-person or over the phone with students participating in 
HL-SCI programs of study. The purpose of the interviews and focus groups was to 
gather student feedback on implementation. 


The data collected from the interviews and focus groups were designed to answer the 
following research questions on program implementation and student success: 


1. Why did the student select the particular program in which s/he is 
participating? 


2. How have the booster modules, online and hybrid courses, and PLAs been 
implemented to support students and their learning? 


3. What steps can the colleges take to improve students’ experiences with booster 
modules and online and hybrid courses? 


4. Did the Consortium colleges make students aware of opportunities to earn 
credit for prior learning? To what extent do students receive PLA credits they 
can use toward completion of a degree or certificate? 
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5. Did students receive career guidance or placement services? If so, how did 
these services support their ability to secure a program-related internship or 
permanent employment? 


6. What were the most important factors contributing to students’ progress 
through and success in their HL-SCI programs of study? 


7. What are the main barriers students must overcome to complete their 
programs successfully? 


CNA collected feedback from students through focus groups and _ interviews 
throughout the grant period, alternating the campuses at which data were collected. 
Researchers spoke with students at Capital, Gateway, and Middlesex Colleges in the 
fall of 2014 and the spring of 2016. Students at Manchester and Norwalk Community 
Colleges provided feedback in the spring/summer and fall of 2015. CNA structured 
the timing of the interviews to ensure that we received feedback from students at 
each college during both a fall and a spring semester. 


The evaluation team worked with grant staff to ensure that the sample of focus 
group and interview participants reflected a wide range of student views. 
Researchers spoke with program coordinators at each college to find out which 
programs they considered particularly important to include in the sample of 
students invited to participate. Priority programs included those that had undergone 
significant revision, were a focus of the grant, or enrolled a large number of students. 
A stratified random sample was drawn that included representation from each of 
these programs.’ 


In the initial round of recruitment, between 25 and 50 students from the stratified 
random sample at each campus were invited to participate in the focus groups. The 
focus groups were scheduled over multiple days during times when students were 
likely to be on campus (for example, before or after times when classes were offered). 
All recruited students received an email invitation, up to two follow-up e-mails, and 
up to two follow-up phone calls. Email invitations were sent to both personal and 
college email addresses to increase the likelihood that the students would receive 
and read them. Lunch and refreshments were provided to incentivize students to 


* Despite the above recruitment efforts, the number of participants at one of the campuses was 
initially quite low. To increase participation, the curriculum innovation coordinator at the 
college reached out to faculty members, who invited their students to contact the evaluation 
team if they were interested in participating in an interview. These students were not selected 
at random so they may not be as representative of the larger sample from which they were 
drawn, but it provided us with much-needed feedback from students at this campus. Overall, 
the findings from the interviews and focus groups at this campus were similar to those from 
the other campuses, which suggests that the use of the nonrandom sample did not bias the 
results. 
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participate. If students were unable to attend a focus group, they were offered the 
opportunity to participate in an on-campus interview at another time or a phone 
interview at their convenience. A second round of recruitment was added for 
campuses with low response rates. 


Some of the grant staff at the Consortium colleges also made concerted efforts to 
increase student participation in interviews and focus groups. For example, the 
curriculum innovation coordinator (CI) at one college emailed all participants to let 
them know that they might be contacted by the evaluators, and also contacted the 
program coordinators to ask them to tell their students to participate in the 
evaluation if contacted. Grant staff also engaged in additional advertising and 
promotion through activities such as posting or handing out flyers about the focus 
groups. 


In total, 167 students across the five colleges provided feedback during the grant 
period. Table 3 shows the total number of students participating in interviews and 
focus groups at each college. Focus group and interview findings were similar across 
semesters. 


Table 3. Focus group and interview respondents at each campus, by semester 


College Fall Spring and Fall Spring Total 
2014 summer 2015 2015 2016 participants 

Capital 8 - - 22 30 
Gateway 6 - - 27 33 
Manchester - 6 26 - 32 
Middlesex 5 - - 26 31 

Norwalk - 16 25 - 41 

Total 19 22 51 75 167 


Challenges associated with the methods used for student recruitment and lack of 
student interest led to low response rates during the first two data collection cycles. 
The research team and Consortium colleges undertook extensive efforts to increase 
the number of students participating in focus groups in the fall of 2015 and spring 
of 2016. The results of these efforts can be seen in the greater number of 
participants in these semesters. 


HL-SC | student survey 


CNA administered a student survey to all HL-SCI participants in the spring of 2015 
and in the spring of 2016. This report includes the results from the 2016 
administration of the survey, which are similar to the 2015 results. The purpose of 
the student survey was to gather feedback similar to that of the focus groups and 
interviews but from a larger group of students. In March of 2016, CNA sent a link to 
the survey to 2,314 students who were currently enrolled in HL-SCI programs of 
study. Of those, 14 percent (N=335) opened and responded to at least part of the 
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survey. Participation was voluntary; grant funding could not be used to provide 
financial incentives for participation. 


Of the 335 students who completed at least part of the survey, 96 percent identified 
their college (N=320). Capital and Norwalk had the highest response rates—20 and 22 
percent, respectively. Middlesex had the lowest response rate. Approximately 7 
percent of the potential respondents completed the survey and identified their 
college (table 4). Thus, although the student survey provides valuable feedback from 
students, it may not be representative of all HL-SCI participants. 


The overall response rate of 14 percent in spring 2016 was higher than in the spring 
2015 administration, which had a response rate of 11 percent. The increase in 
participation resulted from efforts taken by grant staff. The grant’s CIs encouraged 
students to participate in the survey through activities such as emailing participants 
directly with the link to the survey, and posting flyers with a shorted link to the 
survey. Some CIs talked directly with students about the importance of providing 
their feedback through the survey or asked other college staff or instructors to 
discuss this with their students. 


Table 4. Number of student survey respondents and response rates, by college 


College Total students Number of Response rate 
receiving the survey respondents 

Capital 435 87 20 

Gateway 608 70 12 

Manchester 327 34 10 

Middlesex 543 40 7 

Norwalk 401 89 22 

College not identified - 15 N/A 

TOTAL 2,314 335 14 


The number of questions on the spring 2016 survey was reduced from the prior year 
to encourage more students to complete the survey. The survey began with a series 
of questions about where students were enrolled, the programs in which they were 
participating, the people and sources of information influencing students’ decisions 
to enroll in the college, and their reasons for enrolling. Subsequent sections of the 
survey asked students for detailed feedback on the HL-SCI grant’s core components: 
online and hybrid courses, prior learning assessments, and enhanced career guidance 
and placement services. 


Booster module survey 


Student feedback on Consortium-wide implementation of booster modules was 
collected through a separate, voluntary online survey administered by a Survey 
Monkey link posted at the end of the booster module. HL-SCI staff developed the 
survey for all Consortium students who completed a booster module between the fall 
of 2014 and the spring of 2016. The survey was offered at the completion of each 
booster module. Although no data are available on the total number of students who 
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completed a booster module, 4,792 students had the opportunity to complete at 
least one booster module by spring 2016. Therefore, we estimate that the response 
rate is around 14 percent (685/4,792). 


The survey asked students to rate whether the booster was easy to follow, whether 
the booster increased his or her understanding of the subject matter, how the 
student would improve the booster, and whether or not he or she would recommend 
it to others. 


This report builds on booster survey results presented in earlier reports (Pearson, 
Daponte, & Cruz, 2015a, 2015b, 2016). It presents results by academic semester to 
track changes in student feedback over the duration of the grant period. Results 
from the spring and summer of 2015 are combined because relatively few students 
completed boosters during the summer as Consortium colleges did not offer many 
courses that semester. Table 5 displays the number of students who responded 
during each academic time period. 


Table 5. Number of booster survey respondents, by academic term 


Academic term Respondents Percent of total 
respondents 

Fall 2014 165 24 

Spring and summer 2015 270 39 

Fall 2015 184 27 

Spring 2016 66 10 

TOTAL 685 100 


Approach to the impact evaluation 


The quantitative portion of the evaluation seeks to determine whether the HL-SCI 
grant had an impact on student outcomes such as college persistence, credential 
completion, and credit accumulation.’ Outcomes for HL-SCI participants are 
compared with those of a matched comparison group of students enrolled in the 
same or similar programs at Consortium colleges prior to the start of the grant. The 
impact evaluation also includes separate effectiveness analyses for three components 


> The impact evaluation does not include any student employment outcomes because student- 
level employment data were not available. Grant staff sent a list of HL-SCI participants to the 
Connecticut Department of Labor (CT-DOL), and CT-DOL provided a report with aggregate 
employment outcomes for these participants. However, because of student privacy concerns, 
the employment data could not be linked to individual student records and sent to the external 
evaluators. The impact evaluation reports on actual employment outcomes relative to the goals 
for the HL-SCI grant as a whole, but there is no student-level analysis of the impact of the HL- 
SCI grant on employment outcomes. 
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of HL-SCI: PLAs, booster modules, and online and hybrid courses. An overview of the 
data and methods for the impact evaluation is provided in this section, and 
additional detail is included in the technical appendix. 


Research questions 


Overall HL-SCI grant. To evaluate the overall impact of the HL-SCI grant, we answer 
the following questions: 


4. (College persistence) Do HL-SCI participants persist in their colleges at higher 
rates than students who enrolled in similar programs prior to the 
implementation of the intervention? 


5. (Credential completion) Do HL-SCI participants complete certificate or degree 
programs at a higher rate than students who enrolled in similar programs 
prior to the implementation of the intervention? 


6. (Credit accumulation) Do HL-SCI participants complete more credits than 
students who enrolled in similar programs prior to the implementation of the 
intervention? 


For each of these research questions, we examine outcomes separately after one year 
and two years of program enrollment. See tables Al and A2 in the technical appendix 
for additional information on the outcome variables for the impact analyses. We also 
examine whether there are any differences in outcomes by the category of the 
program of study. 


Prior learning assessments. An important component of HL-SCI is to systematically 
award credit for prior education and work experience through PLA systems. PLAs 
allow students to earn credits for experience outside the college, which should allow 
them to progress through their programs more quickly than if they had to 
accumulate all of their credit hours through new coursework. There is some evidence 
in the research literature to support that PLAs increase the likelihood of credential 
completion and reduce time-to-credential (Council for Adult and Experiential 
Learning, 2010; Hayward, 2012). 


To evaluate the effectiveness of PLAs in the HL-SCI grant, we answer the following 
questions: 


1. (College persistence) Do HL-SCI participants who receive PLA credits persist 
in their colleges at higher rates than participants who do not receive PLA 
credits? 


2. (Credential completion) Do HL-SCI participants who receive PLA credits 
complete certificate or degree programs at a higher rate than participants 
who do not receive PLA credits? 


10 


CNA 


3. (Credit accumulation) Do HL-SCI participants who receive PLA credits 
complete a greater number of credits than participants who do not receive 
PLA credits? 


Booster modules. The purpose of booster modules is to provide remediation for 
students in the context of a college-level course. Booster modules are intended to be 
self-paced and adaptable to a student’s schedule. They are meant to allow students 
to hone essential skills, which may improve academic performance and persistence. 
There is limited research available on supplemental instruction similar to the booster 
modules, and the findings are mixed. Some studies have found no effect of 
supplemental instruction on course performance (DeBord, Aruguete, & Muhlig, 2004), 
or inconclusive evidence (Hodara, 2011). However, other studies have found a 
positive impact of supplemental instruction on students’ post-test scores (Aberson, 
Berger, Healy, & Romero, 2003; Hagerty & Smith, 2005) and course completion rates 
(Jenkins, Speroni, Belfield, Jaggars, & Edgecombe, 2010). 


To evaluate the effectiveness of booster modules in the HL-SCI grant, we answer the 
following questions: 


1. (Course completion) Do students who enroll in courses containing booster 
modules complete courses at a higher rate than students who enroll in 
nonbooster versions of the same courses? 


2. (Course performance) Do students who complete courses containing booster 
modules perform better in those courses in terms of course grades than 
students who complete nonbooster versions of the same courses? 


Online and hybrid courses. Online and hybrid courses are designed to increase 
student access to courses and allow students to do self-paced learning, in many 
cases reducing the time needed to complete a course and in some cases permitting 
“asynchronous” learning in which students can view archived lectures when they 
choose. The research evidence of the impact of online and hybrid learning is 
somewhat mixed, with some studies finding negative impacts on student learning 
outcomes such as course persistence and grades (Xu & Jaggars, 2013) and other 
studies finding positive impacts on direct measures of student learning such as 
standardized test scores and course grades (Means, Toyama, Murphy, & Baki, 2013). 


To evaluate the effectiveness of online and hybrid courses in the HL-SCI grant, we 
answer the following questions: 


1. (Course completion) Do students who enroll in online and hybrid courses 
complete those courses at a higher rate than students who enroll in 
traditional in-person versions of the same courses? 


2. (Course performance) Do students who complete online and hybrid courses 
perform better in those courses in terms of course grades than students who 
complete traditional in-person versions of the same courses? 
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Data sourcesand collection 


The primary source of data for the quantitative part of the evaluation consists of 
databases of student records at each of the Consortium colleges from fall 2009 to 
fall 2015. This data source provides for both HL-SCI participants and comparison 
groups the following information: 


e Basic demographic information. 


e Which college the student is attending, with initial enrollment date and date 
of withdrawal or graduation (if available). 


e Whether the student is enrolled full-time or part-time. 


e The certificate or degree program the student is enrolled in, with date of 
entry to the program and date of program withdrawal or completion. 


These databases also provide transcripts for online and hybrid courses and courses 
with booster modules the students have enrolled in and completed, with course 
performance data such as the number of credits and grades earned. 


We also use data collected through HL-SCI grant monitoring activities. The CIs 
provided lists of courses that contain booster modules at their college by semester, 
as well as lists of online and hybrid courses offered by their college by semester. In 
addition, they provided data on the number of credits HL-SCI participants earn 
through PLAs. 


Analysis of outcomes for overall program effectiveness 


For the analysis of outcomes for overall program effectiveness, the intervention 
group consists of cohorts of HL-SCI participants enrolled in spring 2013 (the first full 
semester of implementation of the HL-SCI grant) to fall 2014. As shown in table 6, 
there are three cohorts of students who were enrolled in the first year of the grant 
(spring 2013, summer 2013, and fall 2013), and their outcomes can be followed for 
up to two full years. There are an additional four cohorts of students who were 
enrolled in the second year of the grant (winter 2013, spring 2014, summer 2014, 
and fall 2014) that can be followed for one full year. The comparison group consists 
of students who were enrolled beginning in fall 2009 up to the year prior to the new 
or revised HL-SCI program of study. A list of the specific programs of study in the 
HL-SCI and comparison groups is provided in table 7. 
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Table 6. Timeline for HL-SCI cohorts 


Grantyear  Cohortentry Tem __Lastdata collection Tem N years of follow-up 
1 Spring 2013 Spring 2015 2 
1 Summer 2013 Summer 2015 2 
1 Fall 2013 Fall 2015 2 
2 Winter 2013 Winter 2014 1 
2 Spring 2014 Spring 2015 1 
2 Summer 2014 Summer 2015 i 
2 Fall 2014 Fall 2015 1 


Table 7. Programs of study in the HL-SCI and comparison groups, by program 


category. 


HL-SCI programs of study 


Comparison programs of study 


Health/medical 


CTtomography - new (C) 

Dental assistant - revised (C) 

Diagnostic medical sonography - revised 
(AS) 

Dietetic and nutrition advising pathway - 
revised (AS) 

Exercise science - revised (AS) 

Exercise science and wellness - revised 
(AS) 

Firefighter 1 and 2 certification academy 
- new (C) 

Fitness specialist - revised (C) 
Gerontology - revised (AS) 

Gerontology - revised (C) 

Group exercise instructor - new (C) 
Health and exercise science - revised 
(AS) 

Health science - revised (C) 
Mammography - new (C) 

Medical assisting - revised (AS) 

Nursing - revised (AS) 

Nutrition and dietetics - revised (AS) 
Occupational therapy assistant - revised 
(AS) 

Ophthalmic medical assistant - new (C) 
Paramedic: limited certification - revised 
(C) 

Paramedic studies: emergency 
management - revised (AS) 

Paramedic studies: emergency medical 
services - revised (AS) 

Paramedic studies: general - revised (AS) 
Physical therapy assistant - revised (AS) 
Pre-dental hygiene - revised (AS) 

Public health advising pathway - revised 


Dental assistant (C) 

Diagnostic medical sonography (AS) 
Dietetic and nutrition advising 
pathway(AS) 

Exercise science (AS) 

Exercise science and wellness (AS) 
Fitness specialist (C) 

Gerontology (AS) 

Gerontology (C) 

Health and exercise science (AS) 
Health science (C) 

Medical assisting (AS) 

Nursing (AS) 

Nutrition and dietetics (AS) 
Occupational therapy assistant (AS) 
Paramedic limited certification (C) 
Paramedic studies: emergency 
management (AS) 

Paramedic studies: emergency medical 
services (AS) 

Paramedic studies: general (AS) 
Physical therapy assistant (AS) 
Pre-dental hygiene (AS) 

Public health advising pathway (AS) 
Radiation therapy (AS) 
Radiography (AS) 

Radiologic technology (AS) 
Respiratory care (AS) 

Surgical technology (AS) 
Therapeutic recreation (AS) 
Therapeutic recreation (C) 
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HL-SCI programs of study 


Comparison programs of study 


(AS) 

Radiation therapy - revised (AS) 
Radiography - revised (AS) 
Radiological science - radiation therapy 
- new (AS) 

Radiological science - radiography - 
new (AS) 

Radiologic technology - revised (AS) 
Respiratory care - revised (AS) 
Surgical technology - revised (AS) 
Therapeutic recreation - revised (AS) 
Therapeutic recreation - revised (C) 


e Veterinary technology - new (AS) 


Science 


Biotechnology - new (AS) 
Biotechnology - new (C) 

Biotechnology - revised (AS) 
Engineering science - revised (AS) 
Environmental science - revised (AS) 
Environmental sciences advising 
pathway - new (AS) 

Environmental science and toxicology - 
revised (AS) 

Environmental science and toxicology - 
revised (C) 

Natural sciencesand mathematics - 
revised (AS) 


Biotechnology (AS) 

Engineering science (AS) 
Environmental science (AS) 
Environmental science and toxicology 
(AS) 

Environmental science and toxicology 
(C) 

Natural sciencesand mathematics (AS) 


Data/ IT 


Administrative assistant: medical option - 
revised (AS) 

Communications networking - revised 
(C) 

Computer engineering technology - 
new (AS) 

Data security specialist - new (AS) 
Electronic health record and coding - 
new (AS) 

Electronic health record specialist - new 
(C) 

Health information management - new 
(AS) 

Health information management - new 
(C) 

Help desk technician - revised (C) 
Medical administrative assistant - revised 
(AS) 

Medical administrative assistant - revised 
(C) 

Medical insurance specialist - revised (C) 
Mobile application development - new 
(AS) 


Administrative assistant: medical option 
(AS) 

Communications networking (C) 
Computer and information systems (AS) 
Computer engineering technology (AS) 
Computer hardware support specialist 
(C) 

Computer security (AS) 

Help desk technician (C) 

Medical administrative assistant (AS) 
Medical administrative assistant (C) 
Medical insurance specialist (C) 
Medical office specialist (C) 

Medical office management (AS) 
Network administrative assistant (C) 


Notes: (AS) indicates associate’s degree programsand (C) indicates certificate programs. 
Noncredit programs were not included in the impact evaluation. 
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ANALYSIS & 


Students in the comparison group are carefully selected to match the students in the 
intervention group, using coarsened exact matching (CEM). CEM allows us to correct 
for selection bias in the estimation of intervention and comparison group outcomes, 
thus strengthening the causal implications of any effect that we do find (where 
selection bias is the possibility that students with specific characteristics are more 
likely to choose or be chosen for program participation). 


As the first step in our CEM analysis, we categorize students into groups, or strata.° 
We use an exact matching process that matches a student in the HL-SCI group to a 
student in the comparison group with the same values on key characteristics, 
including: 


e Student status: 


o New student: a student whose first credit earned is upon entry in 
program of study. 


o Continuing student: a student enrolled in any semester in the year 
prior to the first year in a HL-SCI or comparison program. 


o Returning student: a student who was not enrolled in any semester in 
the year prior to the HL-SCI or comparison program, but who earned 
credit at an earlier time. 


e Program category: health/medical, science, or data/information technology 
(IT). 


e Program duration: 
o Fewer than 20 credit hours. 


20 to 24 credit hours. 


(o) 


o 25 to 29 credit hours. 
o 30 to 36 credit hours. 
o 60 to 64 credit hours. 


o 65 to 69 credit hours. 


° Additional information on the variables used in the matching process can be found in the 
technical appendix in table A4 and the corresponding text. Table A3 provides additional detail 
on the classifications for student status and the number of students in each group. 
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This matching process ensures that the HL-SCI and comparison students are the 
same in terms of amount of academic experience, program content area, and 
program duration. This is important for ensuring that outcomes are comparable 
between the two groups. For example, imagine a scenario where: 


e Intervention: N=10 new students in 20-hour programs and N=10 new 
students in 60-hour programs 


e Comparison: N=10 new students in 20-hour programs and N=20 new 
students in 60-hour programs 


We would not want to directly compare credential completion rates against the two 
groups because there are more students in the comparison group who are in 
programs of longer duration (N=20 versus N=10 in 60-hour programs). The CEM 
process would keep all 20 comparison students in the sample, but these students 
would be weighted by .5 so that the intervention and comparison groups would be 
similar overall in terms of program duration. This allows us to use more of the data 
in the sample than if each HL-SCI participant was matched to a single comparison 
student. 


If the scenario was the same as above but the intervention group also included ten 
students in 10-hour programs, these records would be dropped from the sample 
because there are no students in programs of similar duration in the comparison 
group. However, in our study the comparison group is much larger than the 
intervention group, so most HL-SCI students can be matched to comparison students 
in programs of similar duration and type. 


In our data there are 25 different strata. Observations are dropped from three strata 
that do not include at least one treated student and one comparison student. In our 
analysis, 1,364 of the 1,366 HL-SCI students (99.9 percent) and 5,428 of the 5,461 of 
the potential comparison students (99.4 percent) are matched. In the remaining 
analytic sample, comparison units within each stratum are weighted to equal the 
number of intervention units in that stratum. This weighting process ensures that 
the results are representative of the students who participated in the intervention. 


Since the research questions examine outcomes separately for one year and two 
years after initial program enrollment, separate rounds of matching are conducted 
for students with at least one year of follow-up data and at least two years of follow- 
up data (the maximum time for which outcome data can be collected for the first 
cohort of HL-SCI students). This ensures that outcomes are being compared after the 
same duration of time after initial program enrollment for all cohorts. The total 
sample size is 6,782 students with at least one year of follow-up data and 4,201 
students with at least two years of follow-up data (table 8). 


16 


CNA 


Table 8. Number of intervention and comparison group students in the matched 
sample, by years of follow-up data 


Years of Comparison Intervention Total 
follow-up data group Group N 

Atleast 1 year 5,428 1,364 6,782 
Atleast 2 years 3,592 709 4,201 


After matching, to confirm that the intervention and comparison groups are similar 
to each other, we conduct a balancing condition test—that is, whether, among the 
analytic sample, assignment to the intervention group is independent of the 
covariates. We compare the means of the intervention and comparison groups on the 
following characteristics: 


e Student demographic characteristics: percent female, percent Black, percent 
Hispanic, and age. 


e Test scores: scale scores on the Accuplacer assessment in algebra, arithmetic, 
reading, and sentence skills (for the subset of students with test scores). 


e Student characteristics: number of terms enrolled prior to study entry, and 
percent enrolled full-time. 


e College attended (Capital, Gateway, Manchester, Middlesex, or Norwalk 
Community College). 


We find that the intervention and comparison groups are similar on most of these 
covariates (see tables A5 and A6 in the technical appendix). However, the 
intervention group is slightly younger (average age of 28 years versus 32 years for 
the comparison group) and the distribution of colleges differs somewhat because 
some campuses expanded more rapidly than others after the HL-SCI grant began. 


Finally, we estimate the program effect by comparing the outcomes between the 
treated and matched comparison students in all strata. For all of the research 
questions, we examine outcomes separately after one year and two years of initial 
program enrollment. Our primary estimator is the average treatment effect on the 
treated (ATT)—that is, the difference between the average outcomes of treatment 
group members and the (estimated) average outcomes of treatment group members 
had they not been treated. The latter term is not observable and therefore must be 
estimated. We perform the estimation using weighted regression analyses to control 
for demographic characteristics, student characteristics, program characteristics, and 
college attended variables, as described above. It is important to note that these 
impacts should not be interpreted as causal, as there may be unobserved differences 
between the HL-SCI and comparison groups that are not accounted for in the analysis 
and may influence student outcomes. 
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Analysis of outcomes for HL-SCI grant components 


Prior learning assessments. To evaluate the impact of PLAs for HL-SCI participants, 
we use the same regression models as for the overall impact of program 
effectiveness, but we add a dichotomous variable equal to one if any PLA credits have 
been received by HL-SCI participants and zero otherwise. Data on the number of PLA 
credits received were not available for comparison students enrolled prior to HL-SCI. 


We also look at the distribution of PLA credits by calculating the number and 
percentage of students completing different ranges of PLA credit. The percentage of 
HL-SCI students in our sample with PLA credits was 15.3 percent (N=208) for 
students with at least one year of outcome data, and 12.3 percent (N=87) for 
students with at least two years of outcome data. Among HL-SCI participants who did 
have PLA credits, the number of credits ranged from 1 to 22 with an average of 
approximately 4 credits. We also examine how the number of credits affects student 
outcomes by substituting the dichotomous PLA variable with a categorical variable 
that quantifies the number of credits earned (no PLA credits, 1 to 3 credits, 4 to 6 
credits, more than 6 credits). However, due to the small number of students in each 
of these categories, we can conduct the analyses only for students with at least one 
year of follow-up data. 


Booster modules and online and hybrid courses. For booster modules and online 
and hybrid courses, the research design involves comparing course outcomes (course 
completion and grades) between: 


e An intervention group, consisting of HL-SCI participants who enroll in a 
course with an HL-SCI component. 


e A comparison group, consisting of HL-SCI participants and/or comparison 
students who enroll in the same course in another semester prior to the HL- 
SCI component. 


The sample includes all intervention and comparison group students, including some 
HL-SCI students who entered their program of study in the third year of the grant 
(from winter 2014 to fall 2015). The outcomes for these analyses are measured at the 
end of the term in which the course was taken, so it is not necessary for students to 
have a full year of follow-up as in the previous analyses of the overall impact of the 
HL-SCI grant. This means that the intervention and comparison groups are larger for 
these HL-SCI component analyses because they include all students enrolled through 
fall 2015 regardless of follow-up time available. Additionally, students who enrolled 
in multiple courses with an HL-SCI component have a separate record for each 
course, so they are included in the sample multiple times. 


The research design involves comparing an intervention group of HL-SCI participants 
who have enrolled in an online or hybrid course with a carefully constructed 
comparison group consisting of students who enrolled in a prior year in a non-online 
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or hybrid version of the same course. The comparison group is constructed using 
coarsened exact matching to require an exact match on course number, and we 
develop an ATT estimator for the difference in outcomes between intervention and 
comparison group members. The booster analysis includes 8,559 students who took 
the course before the booster was added and 1,038 students who took the same 
courses after the booster was added. The online and hybrid course analysis includes 
5,310 students who took the course in traditional in-person format, and 2,098 
students who took the same courses in online and hybrid formats. 
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Implementation Evaluation Results 


This section presents the results of the implementation evaluation and reports on 
actual performance relative to grant goals for HL-SCI deliverables. Each subsection 
includes figures showing the extent to which the Consortium met the goals of the 
grant at the end of the TAACCCT grant term, a summary of related focus group and 
interview results, and the results of student surveys. The program evaluation results 
related to implementation and performance relative to goals are organized as 
follows: 


e Development and revision of certificate and degree programs. 
e Enrollment and recruitment. 

e Booster modules. 

e Online and hybrid courses. 

e Prior learning assessments. 

e Internship and job placement services. 


e Overall student feedback. 


Development and revision of certificate and 
degree programs 


Two primary goals of the HL-SCI grant were to develop 15 new health and life 
science certificate and associate’s degree programs and to revise 34 existing 
programs by the end of year 3 of the grant.’ HL-SCI also established a goal to 
ensure that 30 of the programs offered are considered stacked / latticed credentials. 
This means that the programs are developed so that a sequence of credentials can be 


” The Consortium developed 22 new programs, but two were duplicated across campuses so 
progress toward the goals includes only the 20 new programs that were nonduplicated. The 
revised programs were revised in a unique way at each campus, so the total includes some 
programs of study that were revised at multiple campuses. 
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accumulated over time to build up an individual’s qualification. To achieve this goal, 
the grant staff followed the definitions of stacked / latticed credentials from the U.S. 
Department of Labor, and any programs that were identified in these categories were 
verified by the Consortium’s Project Assistant, Assistant Project Director, and Senior 
Director. A program was classified as stacked if there was an articulation agreement 
in place that Eastern Connecticut State University cataloged for the Consortium 
indicating that the program was aligned with a more advanced credential. A program 
was Classified as latticed if it could be transferred to another similar program of 
study with very little loss of initial coursework. The Consortium exceeded all three of 
the goals related to the development and revision of certificate and degree programs. 
In total, 20 new, nonduplicated programs and 48 revised programs have been 
implemented across the Consortium. Of these revised programs, 44 are stacked / 
latticed credential programs (figure 1). 


Figure 1. Number of new, revised, and stacked / latticed credential programs 
implemented at Consortium colleges exceeded goals (shown in blue) 


34 
30 
15 
New Programs Revised Programs Revised Stacked / 
(Total) Latticed Programs 


Source: Consortium annual reports to the U.S. Department of Labor. 


Enrollment and recruitment 


The HL-SCI Consortium exceeded its goal of enrolling 600 students in new programs 
by the conclusion of the grant period. By the spring 2016 semester, 637 unique 
students had enrolled in new HL-SCI programs of study at the five colleges (figure 2). 
The HL-SCI Consortium also greatly exceeded its goal of enrolling 2,700 students in 
revised programs by the conclusion of the grant. As of the spring 2016 semester, 
4,371 students are participating or had previously participated in revised programs. 
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Figure 2. Number of participants in new and revised HL-SCI certificate and degree 
programs exceeded goals (shown in blue) 


2,700 


600 
New programs Revised programs 


Source: Consortium annual reports to the U.S. Department of Labor. 


Focus group findings on program recruitment and 
selection 


The first research question for the implementation evaluation addresses why 
students selected the particular program in which s/he is participating. We find that 
most students learn about the HL-SCI programs of study on their own, either through 
an online search or by word of mouth from friends or family members. A few 
students found out about the HL-SCI programs of study because they enrolled in 
other courses at the college and subsequently gained interest in the health and life 
science fields of study. Still others learned about the programs from browsing their 
colleges’ catalogs. Occasionally, students were referred to the program by 
professionals in their fields. For example, staff members at Hartford Hospital 
referred three students to the Radiologic Science program at Manchester. Similarly, 
students in the firefighting program at Gateway and the paramedic program at 
Capital received information about the program from professionals in their fields. 


During each interview cycle, a number of students learned about the programs 
through direct contact with someone at their colleges. Some participants reached out 
directly to program advisors or faculty members, while others attended open houses 
or other recruitment events. A few students were unsure of which program of study 
to pursue and spoke with college advisors, who assessed their interests and 
recommended the HL-SCI programs of study as possibilities. 


Students’ reasons for enrolling 
Across semesters and college campuses, several common reasons emerged for 


enrolling in programs. Students frequently mentioned their interest in or passion for 
their fields of study as a reason. Many students also spoke about the importance of 
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switching careers or using their certificates or degrees to advance in their current 
work settings. 


Others cited the reputation or quality of the program of study at their college. A 
student studying to be a paramedic at Capital said, “I like that it has a good 
reputation. People have told me that the classes are a lot harder than the test [to be 
certified to become a paramedic]. If you do well in the classes, you’ll do well on the 
test.” Another student, studying radiology at Middlesex, said, “It is a highly reputable 
program, one of the best in the state.” 


For many students, affordability and convenience to their homes or jobs were 
important factors in enrolling. Students said the tuition was “attractive” and “cost- 
effective” compared with those of four-year institutions or for-profit options. Some 
students considered four-year or private options, but determined they were too 
expensive and/or far away from home. Other students saw their colleges as the most 
convenient option because they were already employed nearby. 


Guidance on choosing a program 


Each semester, a number of students said they received guidance from program 
advisors or faculty on deciding which program aligned best with their professional 
goals and which courses to take. Most students who received such guidance found it 
helpful. However, some would have liked access to additional advisors with in-depth 
knowledge of HL-SCI programs of study from the beginning of their studies. For 
example, one student at Norwalk said she was initially confused about which 
program would best fit her needs and would have liked more “hands-on” assistance 
from an advisor. 


Student survey findings on enrollment and recruitment 


More than three-quarters of the students who took the survey are enrolled in 
associate’s degree programs. Of the 323 students who indicated which type of 
credential they are pursuing, 83 percent (N=268) are pursuing an associate’s degree. 
The remaining 17 percent indicated that they were enrolled in certificate programs. 


Factors influencing enrollment 


Two questions on the survey focused on the factors that influenced students’ 
decisions to enroll (figure 3). The first question asked about the extent to which 
various people and sources of information contributed to students’ decisions. 
Notably, a third of students indicated that staff or faculty members at their college 
were “extremely influential” to their decision to enroll. An additional 18 percent 
indicated that faculty or staff members were “moderately influential.” College 
websites and catalogs also influenced students’ decisions to enroll. Half of students 
reported that the website was extremely or moderately influential to their decisions 


23 


CNA 


ANA 


to enroll, while 44 percent indicated that the catalog was extremely or moderately 
influential. Less than a quarter of students (23 percent) said that a recruitment event 
was extremely or moderately influential to their decisions to apply or enroll. 


Figure 3. Factors influential or important to student survey respondents’ decisions to 
enroll 


=% Extremely influential or important 
% Moderately influential or important 


People and sources of information 


Staff or faculty at college 18 


College catalog 24 OF, 
College website 20 24 
Print or media advertisement [ile 17 
Recruitmentevent [il) 11 


Staff at CT Works Center 9 


Reasons for enrolling 


Interested in subject of major 2 


Wanted to start a new career ee 
Tuition was affordable fos 15 


Location was close to home or job Ss 19 
Schedule of classes was convenient 19 


Source: HL-SCI student survey, spring 2016. 


Next, the survey presented students with a series of possible reasons for choosing to 
participate in their programs. They were asked to rank on a 5-point scale how 
important each reason was to their decision to enroll. Nearly all students (96 percent) 
cited their interest in their majors as extremely or moderately important to their 
decisions to enroll. 


Booster modules 


As part of the grant, faculty members at Consortium colleges developed booster 
modules, which are incorporated into coursework across the curriculum as 
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appropriate. Boosters, offered through the online learning management software 
BlackBoard, may provide either supplemental course material or a review of content 
that students have previously learned. Some professors require students to complete 
boosters, while others provide them as supplemental assignments or additional 
instructional resources. 


Boosters often contain Microsoft® PowerPoint presentations of lectures and 
multimedia content, such as videos from the Khan Academy or other websites. All 
boosters are designed to include opportunities for students to assess their skills, 
including a pretest, a post-test, and quizzes. Each booster is designed to be 
completed in one sitting, with the range of intended length from 30 to 75 minutes. 


Boosters completed and in development 


Booster modules were offered to students beginning in fall 2014. The Consortium’s 
goal was to develop and implement 140 booster modules across the Consortium by 
the conclusion of the grant period. The Consortium exceeded this goal during years 2 
through 4 of the grant. In total, 154 boosters have been developed and implemented 
during the grant period (figure 4). 


Figure 4. Number of math and science booster modules exceeded goals (shown in 
blue) in most years 


140 


90 
60 


30 


Year1 Year 2 Year 3 Year 4 


Source: Consortium annual reports to the U.S. Department of Labor. 


In addition, HL-SCI staff aimed to have 3,200 unique students take boosters, and the 
Consortium has greatly exceeded this goal. As of the spring 2016 semester, 4,792 
students have taken at least one booster module. 


In spring 2015, HL-SCI contracted with Microburst Learning, an e-learning company, 
to evaluate and enhance the quality of the booster modules and ensure that they are 
compliant with the Americans with Disabilities Act of 1990. 
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Focus group findings on booster modules 


Throughout the grant period, many of the students who participated in focus groups 
and interviews were unfamiliar with booster modules or had not taken one. Those 
who had taken boosters generally found them helpful and useful. For example, a 
student from Gateway found that the booster modules were “clearer than the book 
{and] helped me understand the material.” The student said she wished there were 
more booster modules available. Another student described how the booster 
modules provided extra time and resources to improve her understanding of the 
material. She said, “It takes a while for me to really comprehend what was taught. 
Getting a booster module helps solidify what I already know so that I can store it in 
my long-term memory.” Other students said that taking a booster module helped 
them prepare for upcoming material in their courses. Therefore, a recommendation 
would be that the colleges consider expanding access to booster modules, given that 
self-reported data indicate that they support student success. 


Occasionally, students said that boosters did not hold their attention, were repetitive, 
or were a waste of time. One student from Middlesex recalled taking the same 
booster module three times, saying “they always give us the microscope booster 
module. It is used too much and repeated. I’ve taken the same microscope booster 
module multiple times across multiple different courses.” Another student thought 
that the booster modules were not necessarily a priority given all of the material he 
needed to learn for his course. He said, “Taking five to ten minutes to worry about 
the [booster module] when I could be doing other things as far as studying made it 
not worth it. I didn’t think it was really helpful. It was informative, but it was not the 
information that I needed at the time.” 


Some students who initially were not familiar with boosters said they would be 
interested in completing boosters if they were offered as part of their courses. 
Others said they would likely complete booster modules only if they were a required 
component of their courses. Another recommendation would be that instructors 
consider requiring students to complete boosters to incentivize students to use them 
as a learning tool. 


Survey Monkey student survey on booster modules 


HL-SCI grant staff used the Survey Monkey website to develop a satisfaction survey 
that students at all Consortium campuses were asked to complete after they took a 
booster. Administration began in fall 2014 and continued through the spring 2016 
semester. 


As of May 2016, 685 students across all five college campuses had completed the 
booster survey. This section of the report updates results from the December 2015 
interim report, disaggregated by academic semester to show any changes in 
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responses over time. The spring and summer 2015 semesters were combined 
because few students took the survey in the summer. In the analysis, the results are 
reported by “academic time period.” Relatively few students took the survey in the 
spring of 2016 because not all booster modules included the link to the survey 
(N=66). Nonetheless, these results are presented separately to highlight the 
responses that were not included in previous reports. 


Total results disaggregated by academic time period 


Table 9 shows the number of students who completed the booster survey during 
each academic time period: fall 2014, spring and summer 2015, fall 2015, and spring 
2016. Fewer students took the booster survey in fall 2015 and spring 2016 than in 
previous semesters because some boosters were taken offline in the fall pending the 
results of the content review and enhancements by Microburst Learning. New and 
revised boosters were administered in the spring, but they did not contain the Survey 
Monkey link. 


Table 9. Booster survey respondents by academic tem 


Academic tem N Percent of total respondents 
Fall 2014 165 24 

Spring and summer 2015 270 39 

Fall 2015 184 27 

Spring 2016 66 10 

Total 685 100 


The survey included a question on how long students spent on booster modules, 
scaled in 30-minute increments (figure 5). During all time periods, more than 80 
percent of students completed the booster modules in an hour or less. As booster 
implementation has progressed, the percentage of students who reported completing 
boosters in fewer than 30 minutes has risen, while that of students who reported 
completing a booster in 30 to 60 minutes has declined. Nearly three-quarters of 
spring 2016 respondents reported that they completed the booster module in 30 
minutes or less. Only 12 percent reported dedicating between 30 and 60 minutes of 
time. Very few students reported that it took more than 90 minutes to complete the 
booster. 


Students were also asked to rate the booster’s level of difficulty. More than one-third 
of respondents indicated that the booster was appropriately challenging, with few 
students describing the boosters as “very easy” or “very hard” (figure 6). Across all 
time periods, more students found the booster too easy than too difficult 
(approximately 41 percent versus 21 percent). These results may be expected 
because the content of the boosters is designed to be remedial to help students who 
need additional academic support. 
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Figure 5. Most booster survey respondents reported that they completed boosters in an 
hourorless 


Fall 2014 (N=164) 


Fall 2015 (N=180) 


see 56% TN 11% 
summer 2015 (N=263) 


Spring 2016 (N=65) 


0% 20% 40% 60% 80% 100% 


@ Less than 30 minutes m 30-60 minutes 


™ 60-90 minutes @ More than 90 minutes 


Source: Math and science booster survey, fall 2014 to spring 2016. 


Figure 6. Most booster survey respondents found the boostersto be appropriately 
challenging 


Fall 2014 (N=164) 


Fall 2015 (N=184) 
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summer 2015 (N=263) 7 
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Spring 2016 (N=65) 
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™ Very easy m= Somewhat easy @ Neither easy nor difficult 
= Somewhat difficult m Very difficult 


Note: Totals may not sum to 100 due to rounding. 
Source: Math and science booster survey, fall 2014 to spring 2016. 
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Overall, students indicated that boosters generally increased their level of 
understanding of a subject. The students were asked the extent to which the booster 
increased their understanding, and responded to four options: “much better,” 
“somewhat better,” “the booster had no impact on my understanding,” and “the 
booster was a waste of time.” 


Across all four academic time periods, more than 80 percent of students indicated 
that they understood the topic the booster covered either “much better” or 
“somewhat better” after completing it, although a larger percentage of students 
selected “somewhat better” than “much better” (figure 7). A relatively small 
percentage of students, between 8 and 13 percent, said that the booster had no 
impact. The percentage of students who reported the booster was a “waste of time” 
rose slightly over the duration of the grant, but remained quite small in the spring of 
2016 (8 percent). 


Figure 7. Most booster survey respondents reported a better understanding of course 
content 


Fall 2014 (N=164) 


Fall 2015 (N=182) 


or 2075 (N-2¢7) se 
summer 2015 (N=267) 


Spring 2016 (N=65) 
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m™ Much better mSomewhat better MNoimpact Waste of time 


Note: Totals may not sum to 100 due to rounding. 
Source: Math and science booster survey, fall 2014 to spring 2016. 


Students were then asked to rate their level of agreement on whether it was easy to 
follow the content of the booster. Between 71 percent (spring 2016) and 89 percent 
(fall 2014) of students agreed or strongly agreed that the booster was easy to follow 
(figure 8). Similarly, most students agreed or strongly agreed that the booster clearly 
defined its learning objectives (figure 9). 
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Figure 8. Most booster survey respondents agreed that the boosters were easy to 
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Source: Math and science booster survey, fall 2014 to spring 2016. 


Figure 9. Most booster survey respondents agreed that the booster objectives were 
clearly defined 
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Source: Math and science booster survey, fall 2014 to spring 2016. 
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The next question asked respondents to indicate which factors could have improved 
the booster. The potential responses included “more multimedia,” “less multimedia,” 
“more chances to quiz myself,” “more PowerPoint presentations,” “less PowerPoint 
presentations,” “more written information,” “less written information,” “more 
information overall,” and “no improvement needed.” Students were allowed to select 
multiple items. Of the 685 total responses across the grant period, 90 percent 
(N=615) answered the question. 


“No improvement needed” was the most common response, closely followed by 
“more chances to quiz myself.” The following responses were the most commonly 
selected, generating more than 10 percent of responses cumulatively during all four 
time periods: 


e “No improvement needed” (37 percent, N=225). 

e “More chances to quiz myself” (35 percent, N=216). 
e “More multimedia” (17 percent, N=102). 

e “More written information” (13 percent, N=82). 


Across academic time periods, the majority of the respondents would recommend 
boosters to other students. Students indicated through an open-ended question 
whether or not they would recommend a booster to a friend. Each survey was coded 
as either “yes” or “no” based on the open-ended responses. The percentage of 
students who would recommend a booster declined slightly after the fall 2014 
semester (figure 10). Although the percentage of students who would recommend a 
booster dropped in the final semester, these results should be interpreted with 
caution due to the small sample size (N=29). 
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Figure 10. Most booster survey respondents would recommend boosters to others 
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Source: Math and science booster survey, fall 2014 to spring 2016. 


Online and hybnd courses 


One of HL-SCI’s goals was to develop and implement a total of 60 new online and 
hybrid courses across the five Consortium community colleges. The Consortium has 
exceeded this goal (figure 11). In total, 71 online and hybrid courses have been 
developed and offered to students. This total includes courses that are offered 
through both online and hybrid formats on the same campus. It also includes 
courses that are offered on more than one campus, in one or more formats. 


Figure 11. Number of online and hybrid courses offered exceeded goals (shown in 
blue) in most years 
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Source: Consortium annual reports to the U.S. Department of Labor. 
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In addition, the Consortium exceeded its goal of enrolling 2,400 students in online 
and hybrid courses throughout the grant period. By the spring of 2016, 3,248 unique 
students had enrolled in at least one online or hybrid course. 


The Consortium’s final goal for online and hybrid courses was for instructors at the 
colleges to develop and implement online modules with feedback for students. A 
module is defined as an online lecture equivalency to a traditional, in-person session. 
Therefore, across the 60 online and hybrid courses, the goal sought to ensure and 
increase the availability for students to assess themselves during an online lecture, 
while at the same time receiving instructor feedback on that assessment. The 
Consortium greatly exceeded this goal in the fourth year of the grant. In total, 789 
online modules with feedback have been developed (figure 12).° 


Figure 12. Number of online modules with feedback exceeded goals (shown in blue) 


in year 4 
450 
338 
226 
113 
Year1 Year 2 Year3 Year 4 


Source: Consortium annual reports to the U.S. Department of Labor. 


Focus group and interview findings on online and 
hybrid courses 


Throughout the grant period and across campuses, most students preferred in- 
person courses to the online and hybrid formats.° Generally, students preferred in- 
person instruction because it allowed for greater interaction with professors. In class, 
they could ask questions of their professors and receive immediate responses. One 


8 Microburst Learning conducted a review of online courses after year 2 of the grant, and as 
part of this review it counted the online modules with feedback. These online modules with 
feedback existed when the courses were developed but were not counted until Microburst 
Learning review, so figure 12 shows no online modules with feedback in the first two years. 


° During the focus groups, students were asked about their experiences with any online and 
hybrid courses at the college. The responses may include feedback about some online and 
hybrid courses that were not developed as part of the TAACCCT grant. 
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participant said, “I’m an old-school student. I like to sit in front of the class and raise 
my hand if I have a question. I don’t feel I can teach myself a subject.” Some students 
liked the in-person interactions with instructors because it helped motivate them. 
They also appreciated how their classroom professors often went above and beyond 
their normal teaching duties by providing academic and social support outside of 
class hours. 


Some students felt that the learning that took place during their online and hybrid 
courses was less authentic or challenging than in in-person courses. One participant 
reported learning more during in-person courses, particularly during assessments. 
The student said, “Testing is the biggest thing since if you take a test online, you can 
cheat with a book next to you. With an in-person test, you have to know it.” Other 
students complained that the professors’ expectations for online courses were not 
clear. One student said, “It’s difficult to follow the course guidelines and instructions 
for the hybrid and online courses. It took three sessions to figure out what was 
expected.” 


A few focus group and interview participants experienced technological problems 
with the computer software or hardware required for their online and hybrid 
courses. In some cases, students had difficulty meeting the requirements of online 
and hybrid courses because their personal computers did not have the appropriate 
technological capabilities. For example, their operating systems were not compatible 
with the software required for online and hybrid courses or were too slow to run all 
of the programs required to the extent necessary. In addition, a few students had 
difficulties submitting assignments or quiz responses because BlackBoard did not 
allow for different response formats. 


Although most students expressed a preference for in-person courses, some also 
appreciated online and hybrid courses because they were convenient for students 
who had busy schedules and/or lived far away from campus. Students also liked that 
they could cover course material at their own pace. The flexibility was especially 
important for students who were balancing class schedules with full-time jobs or 
family responsibilities. One student said, “I like going at my own pace. I can knock 
out 80 percent of the class early in the semester and leave myself more time to work 
on other courses later in the semester.” A few students said that the online format 
allowed them to take more courses in one semester than their schedule would 
generally allow. 


In addition, the relative lack of interaction in online and hybrid courses was not an 
issue of concern for all participants. Some students praised their online and hybrid 
course instructors for being particularly helpful or responsive. A number of students 
said that their instructors created in-person lab sessions or offered office hours to 
increase interaction between faculty and students. A few students said that online 
and hybrid courses actually created opportunities for collaboration and interaction. 
For example, a professor in an online Mobile Application Development course graded 
students on the extent to which they interacted with each other through an online 
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discussion board. One of this professor’s students said, “I think that it’s important in 
my field...or any other computer science field [because] you are always working as a 
team. Sometimes in the global economy, team members might not even be in the 
same country as you, so it is important to learn those skills to work with other 
people in an online forum.” 


Students generally believed that the online and hybrid formats were more 
appropriate for some subjects than for others. Students said that courses that 
require either a great deal of explanation or hands-on instruction, such as biological 
sciences, should be taught in-person. According to one student, “it is more difficult 
for students to do science in a hybrid [format]. It’s not great; it’s cheaper, but it’s not 
good for students.” Students said that the online and hybrid course formats may be 
especially appropriate for information technology courses. 


Student survey findings on online and hybrid courses 


The majority of survey respondents (64 percent of a total N of 300) had participated 
in at least one online or hybrid course as part of their studies (figure 13).!° A third of 
students had taken either one or two courses, while nearly a quarter said they had 
taken more than three or more online or hybrid courses. 


Figure 13. Most student survey respondents had taken at least one online or hybrid 
course. 


Number of online and hybrid courses 


™One course MTwocourses MThree courses More than three courses 


64% 


0% 20% 40% 60% 80% 100% 


Source: HL-SCI student survey, spring 2016. 


Students who had taken online and hybrid courses were asked to provide additional 
feedback on their experiences. The survey included a question about how often 


© As with the focus groups, students were asked about their experiences with any online and 
hybrid courses at the college. The responses may include feedback about some online and 
hybrid courses that were not developed as part of the TAACCCT grant. 
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respondents accessed course material from different locations, based on a 4-point 
scale: “very often,” “often,” “sometimes,” or “never.” Most students (92 percent) 
accessed course materials from home very often or somewhat often. Far fewer 
accessed materials from college computer labs or the college library very often or 
somewhat often (33 percent each). 


yb 


Expenencesin online and hybnd courses 


Students who reported taking at least one online or hybrid course were asked to rate 
on a 5-point scale their agreement with statements about their experiences in these 
courses (figure 14). Overall, the feedback on online and hybrid courses was positive, 
with more than 80 percent of students agreeing or strongly agreeing that it was easy 
to access course materials, the flexible schedule made it easier to complete 
assignments, they knew how to ask the professor for help, and it was easy to 
understand their progress in the course. Nearly two-thirds (66 percent) would 
recommend online or hybrid courses to other students. However, one disadvantage, 
according to more than half of student respondents, was that students in online and 
hybrid courses do not interact with each other much. 


Figure 14. Student survey respondents’ level of agreement with statements about 
online and hybrid courses 


= % Strongly Agree or Agree 
% Neutral 
= % Disagree or strongly disagree 


Statement 


It was easy to access online course 
f 87 4 
materials. 
The flexible schedule made it easier 
| knew how to ask the professor for 
help eG 


It was easy for me to understand my 


course(s). 


| would recommend online or hybrid eee ace ees | 20 14 
courses to others students. 

The professor gave me helpful io BB 
feedback on my work. 

Students in online courses do not 25 
interact much. 

| learned as much in my online or 24 
hybrid course(s) as | would in 

courses delivered completely in the 

classroom. 


Source: HL-SCI student survey, spring 2016. 
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Open-ended survey feedback on online and hybrid courses 


The survey included an open-ended item asking students for additional suggestions 
to improve online and hybrid courses. Thirty-five students answered the question, 
and their responses were consistent with the feedback received through interviews 
and focus groups. The most common suggestion was to increase the interaction in 
online and hybrid courses, particularly between students and professors. Some 
students would also like additional opportunities to interact with other students in 
their classes. Two students suggested that instructors in purely online courses hold 
dedicated virtual office hours, time set aside for professors to answer student 
questions by email. 


A few students said that professors did not post course materials or provide 
feedback in a timely way. They would like their online and hybrid course instructors 
to be held accountable for doing so. Three students mentioned experiencing 
technology issues in on-campus computer labs. One said the Internet browser 
crashed during exams, while another said that students are not permitted to 
download software required for online and hybrid courses on campus computers. 
Therefore, many students brought laptops from home to complete online and hybrid 
content on campus. 


As in focus group and interview responses, several survey respondents believed that 
the online and hybrid formats were more appropriate for some subjects than for 
others. Some students felt that they missed out on opportunities for hands-on and 
interactive learning. One student perceived offering science courses in a hybrid 
format as a cost-saving measure. He or she wrote, “I am currently taking a hybrid 
class in pharmacology, and it is a nightmare. It takes at least five times the amount 
of time to complete lectures and notes as an in-person class does. It is not a good 
format for science.” Other students said they felt that they had to teach themselves 
science content offered through an online or hybrid format. Some students indicated 
that these formats worked best for data/IT programs of study. 


Prior leaming assessments 


In collaboration with the Connecticut Career Assessment Program (CCAP), 
administered by Charter Oak State College, the Consortium colleges refined PLA 
systems to award credit for relevant credit and noncredit coursework, prior training, 
and work experience. Charter Oak worked with the Consortium colleges to 
coordinate and standardize the PLA process at each college. For example, Charter 
Oak helped to revise the PLA process at one college from a tedious, paper-based 
portfolio to a fully online application and upload process for students. 


At the outset of the grant, the Consortium set goals of awarding 675 students prior 
learning credit and awarding 10,000 credits throughout the grant period (figures 15 
and 16). The Consortium has exceeded both goals by large margins. By the end of the 
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grant period, the five colleges awarded 15,164 credits to 1,629 unique students 
(figures 15 and 16). This averages out to approximately nine credits per student, 
although the impact on the remaining number of credits needed for credential 
completion varies by credential type and program of study. HL-SCI certificate 
programs require 8 to 36 credits and associate’s programs require 60 to 69 credits. 


Figure 15. Cumulative number of students receiving PLA credits at the conclusion of 
each grant year exceeded goals (shown in blue) 


675 
506 
338 
169 


Year 1 Year 2 Year 3 Year 4 


Source: Consortium annual reports to the U.S. Department of Labor. 


Figure 16. Cumulative number of PLA credits awarded to students at the conclusion of 
each grant year exceeded goals (shown in blue) 


10,000 
7,500 
5,000 


2,500 


Year1 Year 2 Year 3 Year 4 


Source: Consortium annual reports to the U.S. Department of Labor. 


As part of their partnership with Charter Oak, each of the Consortium colleges, as 
well as area noncredit training programs and agencies, has identified noncredit 
credential programs that are eligible for college credit. Charter Oak worked with the 
colleges to identify the applicability of these noncredit programs to certificate and 
degree programs, as well as the number of credits for which students who complete 
the programs are eligible. In 2014, Charter Oak staff met with grant staff at each 
college to streamline the review of noncredit programs at each community college. 
Additionally, Charter Oak promoted the ability for area noncredit training programs 
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and agencies to apply for program review and awarding of college credit for their 
programs. 


As a result of its partnership with Charter Oak, the Consortium has exceeded its 
deliverable goal of completing the credit recognition process for 36 training 
programs by the end of the grant period. By the end of the grant period Charter Oak 
had reviewed and granted credit for 57 programs (figure 17). 


Figure 17. Number of programs processed in the Connecticut Career Assessment 
Program (CCAP) exceeded goals (shown in blue) 


36 36 
24 


12 


Yeari Year 2 Year3 Year 4 
Source: Consortium annual reports to the U.S. Department of Labor. 


The Consortium also set a goal of working with Charter Oak to recognize 324 credits 
from noncredit training programs and high school career and technical education 
programs that can satisfy requirements for college credits. The Consortium exceeded 
this goal by year 2 of the grant and exceeded it by more than twofold in the 
subsequent two years. As of the spring of 2016, Charter Oak has recognized 719 
credits through the CCAP (figure 18). 


Figure 18. Cumulative total credits recognized by Charter Oak through the 
Connecticut Career Assessment Program (CCAP) exceeded goals (shown 


in blue) 
324 324 
216 
108 
Year1 Year 2 Year 3 Year 4 


Source: Consortium annual reports to the U.S. Department of Labor. 
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Focus group and interview findings on prior leaming 
assessments 


Generally, students were aware of the process for awarding prior learning credits, 
but most students who participated in the focus groups and interviews did not apply 
for prior learning assessment nor receive prior learning credit. Of students who did 
participate in the PLA process most generally described the process as 
straightforward and believed they received the appropriate amount of credit for their 
prior experience or knowledge. Among students who attempted to obtain PLA credit 
through an exam (such as CLEP exams or Credit by Exam), a few said that they had 
little information about the format or content of the exams prior to taking them. 


Students’ perceptions of transfering pnor leaming credits and 
academic credits 


Most students said that transferring credits from other institutions was easy and 
convenient. In addition, many students perceived the process of transferring credits 
as fair. Some students, however, fought to ensure that certain credits transferred 
when they believed they had the requisite training and experience. One student, for 
example, experienced challenges with receiving credit for his previous coursework 
and work experience in the military. He said, “They were leery of accepting two 
advanced chemistry courses that I took at [previous college] because they were more 
than five years old. I had to petition for their acceptance by giving them my resume 
[which included] work experience heavy in a chemistry background...I had to go 
through a tortuous process with [my college] for them to overlook that my chemistry 
classes were outside of a five-year timeframe.” Another student had completed a 
bachelor’s degree in psychology prior to enrolling in her program, but had to take 
developmental psychology again because the course names at the two institutions 
did not align. 


Other students perceived the process for transferring credits as administratively 
burdensome or were unsure of the criteria that their colleges used to award credit. 
Some students said that the process for transferring credits took a long time, 
requiring multiple email exchanges and in-person visits to the college. One student at 
Norwalk was enrolled in a physical therapy program, but staff members mistakenly 
thought she was a nursing student and initially did not award her the correct number 
of transfer credits. She appealed their decision, but it took three months for them to 
award credits for her prior coursework. Several international students experienced 
challenges with attempting to transfer credits from foreign institutions. These 
students often paid substantial fees for translating foreign transcripts, which 
presented a financial challenge. 


A few students would have preferred a more personalized process for determining 
whether students should receive prior learning credit. For example, one student said 
that the process for awarding credit should be based on students’ familiarity with 
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the course material and their abilities to apply what they have learned to subsequent, 
more advanced coursework. The student said, “They should take more time in 
getting an overall picture, rather than going line by line.” Students said that program- 
specific advisors often knew more about which credits would be appropriate to 
transfer into HL-SCI programs of study than collegewide advisors. They often 
preferred meeting with advisors within their department or major. 


Student survey findings on prorleaming assessments 


The section of the student survey on PLA opened with a question about whether 
students had received prior learning credit. Of the 285 students who answered the 
question, 29 percent said yes, 40 percent said no, and 31 percent were unsure. Only 
those students who answered yes were asked subsequent questions on PLAs. 


These students were asked how many credits they had received through the PLA 
process that counted toward their certificate or degree program. Only 11 percent of 
respondents said that none of their PLA credits counted toward their programs, 
while nearly 40 percent said they received more than 12 credits to fulfill degree 
requirements (figure 19). 


Figure 19. Most student survey respondents who participated in the PLA process 
received credits that were applicable toward theirdegrees 


Number of degree-applicable PLA credits received 
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Source: HL-SCI student survey, spring 2016. 


Because of the highly applied nature of the coursework and courses in the HL-SCI 
programs of study, students are not able to transfer credits for many of their 
required courses. Additionally, many of the outside accrediting bodies of the HL-SCI 
programs specifically prohibit transferring of credits toward the required core 
courses. However, depending on the PLA credits awarded, students may be able to 
apply those credits to general education requirements for their degrees. 


41 


CNA 


As in the section of the survey on online and hybrid courses, students who had 
received prior learning credits were asked to indicate on a 5-point scale the extent to 
which they agreed with a series of statements on the PLA process (“strongly agree,” 
“agree,” “neither agree nor disagree,” “disagree,” and “strongly disagree”). Table 10 
displays the statements, as well as the number of students who responded to each, 
the percentage who agreed or strongly agreed, and the percentage who disagreed or 
strongly disagreed. 


Table 10. Student survey respondents’ level of agreement with statements about prior 
leaming assessments 


Statement Agreeing or Disagreeing 
strongly or stongly 
agreeing disagreeing 

The process for receiving prior leaming credit was easy 771% 10% 
to understand. 

| think | received the rght amount of credit for my prior 71% 14% 
coursework, training, or experience. 

Asa result of receiving priorleaming credit, | think | will 69% 12% 
complete my certificate orassociate’s degree program 

faster. 

Asa result of receiving prior leaming credit, |am more 60% 11% 
motivated in my courses. 

lam more likely to complete my certificate or 57% 13% 


associate’s degree program because | received prior 
leaming credit. 


Source: HL-SCI student survey, spring 2016. 


Most students believed that the process for receiving prior learning credit was 
straightforward. In addition, more than 70 percent believed that they had received 
the correct amount of credit given their prior knowledge and experience. More than 
half (57 percent) reported that receiving prior learning credit increased their 
likelihood of completing a degree or certificate. 


Intemships and job placement services 


Across campuses, the HL-SCI grant deliverable goals for internships and job 
placement services were to place 360 participants in internships and provide job 
placement services to 2,000 participants. The Consortium has greatly exceeded each 
of these goals. During the grant period, 2,412 unique participants have completed an 
internship and 4,248 unique participants have received employment placement 
services (figure 20). 


A final Consortium goal related to employment and placement services was for 1,350 
students to take online career or soft skills assessments, such as Focuses, Bridges, 
and Career Edge. The definition also included online soft skills assessments with 
instructor feedback that might be offered within specific courses. The Consortium 
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exceeded its deliverable goal. Throughout the grant period, 2,478 unique students 
took an online skills assessment. 


Figure 20. Number of participants placed in intemships and receiving employment 
placement services, and students taking online skills assessments 
exceeded goals (shown in blue) 


2,000 
1,350 


360 


Participants placed in Participants receiving Students taking online 
internships employment placement skills assessments 
services 


Source: Consortium annual reports to the U.S. Department of Labor. 


Focus group and interview findings on intemships and 
employment placement services 


Across semesters and campuses, most of the students who participated in hands-on 
work experiences as part of their HL-SCI programs of study participated in clinical 
practicums. Overall, these students spoke positively about their clinical experiences. 
They appreciated that they were hands-on, reinforced academic content, and allowed 
them to apply what they had learned in the classroom. Students were able to see a 
connection between their clinical experiences and future careers. According to one 
student, “The bulk of my job now is working with patients and understanding how 
they get treated and why we treat them this way, and I believe the clinical portion 
really did prepare us for working in the field.” 


A couple of students said that the professionals supervising their clinical experiences 
had limited time to provide hands-on instruction and guidance. “They’re often busy 
and don’t have time to teach you,” one student said. “You can make observations 
about what is going on, but they don’t have time to explain to you what is 
happening.” In addition, a few students disliked the rigid scheduling of clinical 
experiences, noting that it is especially challenging to participate in practicums and 
hold a full- or part-time job. Students would like more input into both the hours they 
worked and their clinical placement sites. 
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Some students reported receiving career advice or guidance from their colleges, 
including career advice and help with resumes and interview skills. Those who 
received guidance from either program advisors or faculty members generally found 
it useful. Students cited faculty members as especially helpful with finding 
permanent jobs. According to one nursing student, “They are helping us with our 
CVs [curriculum vitae]. The teachers are willing to write us reference letters. If you 
get an interview, the professors are available to help you prepare.” 


Throughout the grant period, some students said they would like additional career 
guidance, including help with securing a job within their fields while they are still in 
school and obtaining permanent employment once they complete. At Capital, focus 
group participants recommended the development of an HL-SCI mentoring program. 
Community members who worked in health and life science fields would serve as 
mentors and help students transition from college to career. 


In multiple interviews and focus groups, nursing students said they would have liked 
more information about the value of an associate’s degree in nursing prior to 
entering the program, since most nursing jobs in Connecticut now require a 
bachelor’s degree in nursing. According to one student at Norwalk, “There needs to 
be more advising for students to figure out how they can get the [B.S.N.] degree. 
There should be mandatory check-ins with our advisors once a semester.” In a 
subsequent semester, a Norwalk nursing student said that faculty had arranged for 
four-year universities to provide information on how to apply for bachelor’s degree 
programs. 


Student survey findings on intemships and 
employment placement services 


The section of the student survey on internships and placement services included a 
multipart question on whether students had received specific types of internships 
and placement services and how helpful they found them. Table 11 displays the 
services, the percentage of respondents who received each service, as well as the 
percentage who found them helpful, unhelpful, or neither. 


Most commonly, students reported receiving assistance with course selection, and 
nearly three-quarters of these students found the advising helpful. More than half of 
the respondents received information on jobs to pursue either while enrolled or after 
completing the program. A majority of students also found these services helpful. 
However, fewer than half of students who received help with resume revision or 
finding a permanent job found these services helpful. The survey section concluded 
with an open-ended question about what students would improve about internship 
and employment placement services. Overall, students would like more employment 
advising. Some students said that they received more effective advising from faculty 
than staff members designated as advisors. Two respondents said that students were 
responsible for finding and securing their own internships in their programs. These 
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students would like additional information about prospective employers prior to 
beginning the application process. Finally, several students who work while attending 
college would like employers to offer internships with more flexible schedules. They 
said that most internships require students to work full-time, but a part-time 
internship would provide them with valuable skills to add to their resumes. 


Table 11. Student survey respondents who received specific placement services and 
their ratings of the level of helpfulness of each service 


Rating of those receiving the service 

Service Percent Very Neither Unhelpful 
receiving helpfulor helpful nor orvery 

the service somewhat unhelpful unhelpful 


helpful 
Assistance with course selection 76% 74% 16% 10% 
Information on jobs to pursue 65% 71% 21% 8% 
after program completion 
Information on jobs to pursue 59% 66% 23% 11% 
during the program 
Help with resume revision 52% 59% 26% 15% 
Assistance with intemship 48% 58% 27% 15% 
placement 
Help with interview skills 47% 45% 41% 15% 
Assistance with job-shadowing or 44% 56% 32% 12% 
another work-based training 
experience 
Assistance with placement ina 40% 40% 42% 18% 
permanent job 


Source: HL-SCI student survey, spring 2016. 


Foc us group and interview findings: overall 
student feedback 


What students liked best about their programs 


On the whole, students provided positive feedback about their HL-SCI programs 
of study. Most interview and focus group participants believed that their 
programs would change their lives by allowing them to secure a well-paying job in 
an in-demand career field. Students said their classes motivated them to achieve 
academically. One student said of her program, “It has shown me that I can do 
this and will give me the direction that I want.” 


Students especially appreciated that much of their course content was offered 
through hands-on instruction in laboratory or clinical settings. In the spring of 
2016, students at Gateway had particularly favorable impressions of the Human 
Cadaver Anatomy course, offered through the biology department, which 
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provided hands-on training that students would not receive elsewhere. According 
to one student, “This class is really awesome! We're working on a real-life human 
cadaver...we’re basically dissecting and looking at parts of the body, and that’s 
awesome.” Another student said, “It is probably the best undergraduate course I 
have ever taken...usually, students with our background wouldn’t have this 
experience.” Several other students made similar comments about the class. 


A fire science student also appreciated the hands-on experience in his courses. He 
said, “We were able to use the New Haven Fire Academy, which offers a lot of 
tools and training. We had access to their burn building, and we had access to 
their tower. We were able to use these for practical exercises.” In previous 
semesters, students similarly praised laboratory and clinical experiences because 
they offered the opportunity to apply what they had learned. 


Students nearly universally spoke favorably about their professors. Students said 
that professors were well-prepared and dedicated to students. Overall, they 
presented the course material clearly and were responsive to questions. One 
student said, “I love my instructors. They go above and beyond to make sure you 
understand what they are teaching, and they don’t mind taking the time out of 
their busy schedules to help you.” Students liked that many of their professors 
either had current or prior work experience in health and life science fields. Thus, 
they were able to relate lectures and labs to on-the-job experiences. According to 
one student, “It’s so very easy to learn from them because they have stories they 
can tell and that makes it easier to learn.” A few students were critical of their 
professors, but their complaints were generally specific to the context of each 
course. There were no consistent criticisms across either time or HL-SCI colleges. 


We spoke with a limited number of students who were veterans, who generally 
provided positive feedback on both their overall experiences and the support 
services designed specifically for veterans. For example, students at Gateway 
praised the veterans’ coordinator at the college and described him as a “problem 
solver.” These students said that he served as a liaison between students and 
faculty and staff members, helping students address any issues that arose. One of 
the Gateway students appreciated the extra services and “community” available 
for veterans at Gateway so much that he or she elected to commute over an hour 
to attend the college. 


Students’ challengesand suggestions for improvement 


Throughout the grant period, the most significant challenge for HL-SCI students was 
balancing their course loads with work, family responsibilities, or both. Some 
students had difficulty arranging their course schedules around other obligations. 
Students suggested offering commonly required courses during both day and 
evening sessions to ensure that as many students as possible can participate. One 
student said that the introductory chemistry and biology courses he needed for his 
degree were scheduled at the same time, which made it challenging to progress 
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through the program at the rate he would have liked. Similarly, a student at Norwalk 
said that it was difficult to get through a new program in a timely manner, since only 
one or two required courses were offered per semester. 


Time management also presented a challenge. Some students had difficulty finding 
time to complete homework and other assignments given their competing demands. 
Some students dropped courses or reduced their hours at work to ensure that 
coursework was completed. 


Some students would like additional access to on-campus facilities to support their 
coursework. For example, students at Manchester would like to extend the library 
hours beyond the 8 p.m. closing time to allow them greater flexibility. Also at 
Manchester, an occupational therapy student cited the need for additional course 
content in skilled nursing and pediatric therapy. The student said, “Other programs 
have pediatric gyms that you can work in, and I never got to see that.” 


Some students requested additional support services from their colleges. During data 
collection cycles early in the grant, students requested additional childcare on 
campus or assistance with childcare expenses. Other students requested assistance 
with nontuition expenses. For example, students would like assistance with paying 
for parking at clinical sites. 
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Impact Evaluation Results 


This section describes the results from analyses assessing the impact of the HL-SCI 
grant on student outcomes. We begin by providing information about the extent to 
which the Consortium achieved its goals for student outcomes. Next, we examine the 
overall impact of the HL-SCI grant on college persistence, credential completion, and 
credit accumulation. Finally, we provide results for the effectiveness analyses of 
three HL-SCI components: PLAs, booster modules, and online and hybrid courses. 


Actual performance relative to student 
outcome goals 


In the original grant application, HL-SCI staff set nine goals for student outcomes, 
and progress toward these goals was monitored in annual reports to the U.S 
Department of Labor. The first set of goals relate to student enrollment and 
completion outcomes by the end of the third year of the grant. The Consortium 
exceeded all of these goals (figure 21). In particular, the number of students who 
participated, remained in their programs, and completed credit hours exceeded the 
goals by more than 1,000.'' The goals for the number of participants completing 
their programs and the total number of credits earned were exceeded by a smaller 
amount, although the majority of HL-SCI participants were still enrolled at the end of 
the follow-up period, so these outcomes are assessed among a much smaller group.” 


" The indicator for the number of participants retained includes the total number of new and 
continuing participants who were retained in each year of the grant. Participants are double- 
counted if they are retained for more than one year. 


”? The number of participants completing programs is a unique count of students who earned a 
credential. This differs from the total number of credentials earned, which double-counts 
participants who completed multiple credentials such as a certificate and an associate’s degree. 
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Figure 21. Performance on student enrollment and completion outcomes through 
year 3 exceeded goals (shown in blue) 


2,700 
2,430 
2,244 
783 861 
Participants Participants New and continuing Participants Total credentials 
served completing participants retained completing earned 
programs in their programs credit hours 
(not unique) 


Source: HL-SCI annual reports to the U.S. Department of Labor. 


The second set of student outcome goals relate to continuing education and 
employment, and are measured four years after the start of the grant. The 
Consortium exceeded two of these four goals (figure 22). First, the number of 
completers who continued their education exceeded the goal (234 versus 196). This 
number includes only completers who attended a public college in Connecticut, so 
the actual number may be even higher if some completers attended a private or out- 
of-state college to continue their education. Additionally, data on continuing 
education were available only after a two semester delay, so the totals do not include 
participants who continued their education at a four-year state institution after fall 
2014. 


Next, the Consortium more than tripled its goal for the number of participants 
employed at enrollment who received a wage increase (1,408 versus 405). This 
number includes all participants (not just completers) who have received a pay raise, 
which may be a result of pursuing new or additional credentials (although some may 
have received a pay raise for other reasons). In total, 69.4 percent of incumbent 
workers had a wage increase in any quarter after HL-SCI enrollment. The average 
annual wage increase was $7,036, but varied by college from $5,397 to $8,789. 
Earnings outcomes for incumbent workers also differed by the category of the 
program of study (table 12). 
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Figure 22. Performance on student continuing education and employment outcomes 
through year four exceeded two of four goals (Shown in blue) 
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Completers enrolled First-time employment Retention in first-time Participants employed 
in further education for completers employment for at enrollment w/wage 
completers increase 


Source: HL-SCI annual reports to the U.S. Department of Labor. 


Table 12. Eamings outcomes for HL-SCI participants who are incumbent workers, by 
program of study 


Total Percent of 

incumbent Participants participants Average 
Program worker with wage with wage wage 
category Participants __increases increase increase 
Health/medical 1,631 1,107 67.9 $7,137 
Science 202 157 77.7 $5,983 
Data/IT 196 144 73.5 $7,461 
All 2,029 1,408 69.4 $7,036 


Source: Employment data provided by the Connecticut Department of Labor. 


The two student outcome goals that the Consortium did not meet are measured only 
for graduates who were not employed at the time of enrollment in HL-SCI. They do 
not consider prior, historical employment nor employment post-enrollment; they are 
a snapshot at time of intake only. These outcomes examine whether these graduates 
gained employment in the quarter following completion of their HL-SCI program, 
then subsequently retained employment for three quarters following initial 
employment. However, there was considerable variation in these outcomes across 
programs and colleges (table 13). The percentage of graduates in post-completion 
employment was highest for science programs of study (68.2 percent), followed by 
health/medical (61.1 percent), and then data/IT (58.5 percent). The percentage of 
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completers employed ranged by college from 53.2 to 78.2 percent. For the outcome 
examining the percentage of graduates retained in employment for three continuous 
quarters after initial employment, the rates were similar (within 3 percentage points) 
for all program categories. There was greater variation in this outcome across 
colleges, where employment retention rates for first-time workers ranged from 28.6 
percent to 62.2 percent (table 14). 


Table 13. Employment outcomes for HL-SCI graduates who were not employed atthe 
time of enrollment, by program of study 


Continuous 
Employment at 1st employment for 3 
quarter after quarters after initial 
graduation employment 
Program category’ Totalgraduates N Percent N Percent 
Health/medical 560 342 61.1% 248 44.3% 
Science 22 15 68.2% 10 45.5% 
Data/IT 53 31 58.5% 25 47.2% 
All 635 388 61.1% 283 44.6% 


Source: Employment data provided by the Connecticut Department of Labor. 


Table 14 Employment outcomes for HL-SCI graduates who were not employed at the 
time of enrollment, by college 


Continuous 
Employment at 1st employment for 3 
quarter after quarters after initial 
graduation employment 
College Total graduates N Percent N Percent 
Capital 255 136 53.3% 73 28.6% 
Gateway 143 102 71.3% 89 62.2% 
Manchester 87 68 78.2% 51 58.6% 
Middlesex 11 8 72.7% 6 54.5% 
Norwalk 139 74 53.2% 64 46.0% 
All 635 388 61.1% 283 44.6% 


Source: Employment data provided by the Connecticut Department of Labor. 


Overall impact of HL-SCI on student 
outc omes 


This section examines the overall impact of the HL-SCI grant on student outcomes, 
including college persistence, credential completion, and credit accumulation. 


a1 


CNA 


Outcomes for HL-SCI students are compared with a matched comparison group of 
students enrolled in the same or similar programs of study prior to the new or 
revised HL-SCI programs of study. The matching process ensures that the 
intervention and comparison students are the same in terms of their prior academic 
experience (new, continuing, or returning student), the content area of their program 
of study (health/medical, science, or data/IT), and the duration of their program of 
study (fewer than 20 credit hours, 20 to 24 credit hours, 25 to 29 credit hours, 30 to 
36 credit hours, 60 to 64 credit hours, or 65 to 69 credit hours). This is done by 
making comparisons among intervention and comparison groups within the same 
stratum, and then aggregating the results across all strata. 


For each student outcome assessed, we provide an illustrative graphical display of 
the results from the regression models using bar charts. These charts display the 
results for two hypothetical populations—one with all HL-SCI students and one with 
all comparison students—that have the same values on all other variables in the 
model, including student demographics, student characteristics, program 
characteristics, and college attended (see technical appendix for more detail). This 
shows how outcomes for the two groups would differ if all other factors were held 
constant. 


The notes at the bottom of each figure provide the p-value, which is used to assess 
the statistical significance of the impact estimate. The p-value represents the 
likelihood of obtaining a finding due to random chance rather than to the 
investigated effect. Using a two-tailed test, we consider an impact to be statistically 
significant if there is less than a 5 percent chance of obtaining a random finding (a p- 
value of less than 0.05). The figures also include an asterisk next to the value for the 
HL-SCI students if the results are statistically significant. Detailed regression results 
are presented in tables A7 and A8 of the technical appendix. 


College persistence 


The first outcome that we examine for the overall impact of the HL-SCI grant is a 
dichotomous variable for whether the student persisted in college one year after 
initial program enrollment (O=no longer enrolled, 1=still enrolled or completed the 
program). For example, if a student enrolled in an HL-SCI program of study during 
spring 2013, we would examine whether the student was still enrolled in college 
during spring 2014 (in any program of study). We also assess the impact of the HL- 


'’ Note that if a student enrolled in an HL-SCI program of study during spring 2013, did not 
enroll in spring 2014, but returned in a subsequent semester, the student is still classified as 
“no longer enrolled” for the year 1 persistence outcome since outcomes are being compared 
after the same duration for all cohorts. However, if the same student was enrolled again in 
spring 2015, the student would be classified as “still enrolled” for the year 2 persistence 
outcome. 
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SCI program of study on continued persistence for two years after initial program 
enrollment for the subset of students with at least two years of follow-up data. 


We find that the HL-SCI grant does not have a statistically significant impact on 
college persistence in either year. The predicted probability of persistence for HL-SCI 
students is 68.9 percent after one year of program enrollment and 66.2 percent after 
two years of program enrollment (figure 23). Persistence outcomes are similar for 
students in the comparison group. 


Figure 23. Predicted probability of persisting in college for students in the comparison 
and HL-SCI groups, after one and two years of program enrollment 
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Years of follow-up after initial program enrollment 


Notes: Results are from a logit model that accommodates for clustering of students within 
colleges (N=6,197 in year 1 and N=3,836 in year 2). The model is used to calculate the 
probability of persisting in college by HL-SCI status, while holding constant all other 
variables. The coefficient on the HL-SCI variable is 0.007 (p =0.938) in year 1 and 0.033 
(p=0.817) in year 2. 


* Coefficient (of estimated intention-to-treat impact) is statistically significant at the 95 
percent confidence level using a two-tailed test. 


We also examine whether there are any differences in the likelihood of persistence by 
program category. The sample sizes are small for some program categories 
(particularly for outcomes after two years of follow-up), which means that the impact 
estimates by program category are subject to lower reliability than the impact 
estimates for the program as a whole. It is important to consider not only the 
differences in the marginal effects between the program categories, but also the 
confidence intervals which provide a range of likely values for the effects. If the 95 
percent confidence intervals overlap between two program categories, then any 
differences may have occurred by chance. 
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Predicted probabilities are calculated by HL-SCI status and program category while 
holding constant student demographics, student characteristics, program 
characteristics, and college attended. We find that in the intervention group, the 
predicted probability of persisting in college for one year and two years is similar 
regardless of program category (table 15). The 95 percent confidence intervals for 
each of the program categories overlap with the confidence interval for all programs 
among HL-SCI participants. In the comparison group, persistence rates are mostly 
similar by program category, except that the science program category has a lower 
persistence rate in year one relative to all programs in the comparison group. 


Table 15. Predicted probability of persisting in college by program categony, afterone 
and two yearsof program enrollment 


Comparison group Intervention group 
Persistence: One year 
95% 95% 
confidence confidence 
N Margin interval N Margin interval 


Health/medical 3,488 70.6% 67.8% 73.4% 856 71.1% 67.9% 74.3% 


Science 539 57.2% 51.2% 63.3% 246 68.8% 62.7% 74.9% 
Data/IT 833. 74.5% 70.9% 78.1% 235 61.9% 55.4% 68.4% 
All 4,860 68.8% 66.7% 71.0% 1,337 68.9% 66.2% 71.4% 
Persistence: Two years 
95% 95% 
confidence confidence 
N Margin interval N Margin interval 


Health/medical 2,218 67.8% 63.5% 72.1% 480 71.9% 67.2% 76.6% 


Science 377 = 552.0% 40.5% 63.5% 105 55.7% 44.8% 66.5% 
Data/IT 539 68.2% 60.5% 75.8% 117 54.9% 45.6% 64.2% 
All 3,134 65.5% 62.0% 68.9% 702 66.2% 62.3% 70.4% 


Notes: Results are from a logit model that accommodates for clustering of students within 
colleges (N=6,197 in year 1 and N=3,836 in year 2). The model is used to calculate the 
probability of persisting in college by HL-SCI status and program category, while holding 
constant all other variables. 


Credential completion 


Next, we examine whether the HL-SCI grant has an impact on the likelihood that 
students will complete a credential after one or two years of program enrollment. 
The primary outcome that we examine is the likelihood of completing any credential 
for all students. However, we also examine the likelihood of completing an 
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associate’s degree for students in programs of study that lead to an associate’s 
degree, and the likelihood of completing a certificate for students in programs of 
study that lead to a certificate. One caveat is that many students who register for a 
certificate program are also registered for the relevant Associate’s degree program. 
Because of the U.S. Department of Labor’s limitations on unique participant coding, 
at time of student intake, each college chose whether to code students as certificate 
or associate-seeking students, with the majority choosing the former. In these cases, 
students would not be shown as completing a certificate until after they graduated 
with an associate’s degree. This may underestimate the number of students 
completing a certificate since there is a limited follow-up period for tracking both 
certificate and associate’s degree completion for those students. Each outcome for 
credential completion is operationalized as a dichotomous variable where a value of 
one indicates that the student completed a credential and a value of zero indicates 
that the student did not complete a credential within the follow-up period. 


In both the HL-SCI and comparison groups, less than 2 percent of students complete 
any credential (certificate or associate’s degree) within one year of program 
enrollment (figure 24). This low rate is not surprising given that most programs of 
study require more credits than students can complete in a single year. Among all 
students in the sample, the average number of credits required for program 
completion is 60.8 credits. That means that a full-time student would not be 
expected to complete their program of study for at least two years (for example, 15 
credits per semester in the fall and spring semesters over two years, for a total of 60 
credits). Time-to-credential will be even longer if students are required to complete 
courses that are not offered every semester, or if they are enrolled part-time (only 
about one-third of students in our sample are enrolled full-time). We find that in our 
sample, the predicted probability of completing a credential after two years of 
program enrollment is 17.1 percent for HL-SCI students, which is not statistically 
different from the comparison group. 


35 


CNA 


ANALYSIS & 


Figure 24. Predicted probability of completing any credential for students in the 
comparison and HL-SCI groups, after one and two yearsof program 
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Notes: Results are from a logit model that accommodates for clustering of students within 
colleges (N=6,218 in year 1 and 3,840 in year 2). The model is used to calculate the 
probability of completing a credential by HL-SCI status, while holding constant all other 
variables. The coefficient on the HL-SCI variable is 0.401 (p=0.180) in year 1 and 0.145 
(p=0.442) in year 2. 


* Coefficient (of estimated intention-to-treat impact) is statistically significant at the 95 
percent confidence level using a two-tailed test. 


Credential completion rates also tend to be similar by program category (table 16). 
Within the intervention group, the 95 percent confidence interval for each program 
category falls within the range of the confidence interval for all programs. In the 
comparison group, students in science programs are less likely to complete a 
credential after two years of program enrollment, holding other factors constant. 


Within our sample, approximately 90 percent of students are enrolled in associate’s 
degree programs. When we estimate the likelihood of completing an associate’s 
degree after one or two years of program enrollment for these students, there are no 
statistically significant differences between the HL-SCI group and the comparison 
group. We also examined credential completion outcomes for the small subset of 
students enrolled in certificate programs. We find a small increase in the predicted 
probability of completing a certificate after one year for HL-SCI students relative to 
comparison students (5.5 percent versus 2.5 percent). However, this difference is 
small in magnitude and is not robust to sensitivity analyses that exclude continuing 
students (see table A8 in the technical appendix for complete results). We also do not 
find any statistically significant differences between HL-SCI and comparison students 
on the likelihood of completing a credential after two years of program enrollment. 
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Taken together, these results suggest that HL-SCI grant does not have an impact on 
credential completion. 


Table 16. Predicted probability of completing any credential by program category, 
afterone and two years of program enrollment 


Comparison group Intervention group 
Any credential: One year 


95% 95% 
confidence confidence 
N Margin interval N Margin interval 
Health/medical 3,493 1.1% 0.9% 1.5% 865 1.8% 0.8% 2.8% 
Science 539 0.5% 0.0% 1.0% 248 1.8% 0.0% 3.6% 
Data/IT 833 1.2% 0.7% 1.7% 240 1.2% 0.1% 2.3% 
All 4,865 1.1% 0.9% 1.3% 1,353 16% 0.9% 2.3% 
Any credential: Two years 
95% 95% 
confidence confidence 
N Margin interval N Margin interval 
Health/medical 2,219 19.4% 15.9% 22.9% 481 20.3% 16.1% 24.6% 
Science 377 0.6% 0.0% 1.3% 106 10.3% 41% 16.5% 
Data/IT 539 10.1% 56% 146% 118 8.8% 3.8% 13.8% 
All 3,135 15.3% 12.9% 17.7% 705 17.1% 13.7% 20.1% 


Notes: Results are from a logit model that accommodates for clustering of students within 
colleges (N=6,218 in year 1 and 3,840 in year 2). The model is used to calculate the 
probability of completing a credential by HL-SCI statusand program category, while 
holding constant all other variables. 


Credit accumulation 


The last outcome that we examined for the overall impact of the HL-SCI grant is 
credit accumulation. This is a continuous variable for the total number of credits 
earned through coursework during the study period. It does not include any credits 
earned for previous coursework for continuing or returning students, or credits that 
were acquired through the PLA process. This outcome is examined after one year and 
two years of initial program enrollment. 
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We find no statistically significant differences in credit accumulation between the HL- 
SCI students and comparison students. The regression-adjusted mean is 
approximately 15 credits earned in year 1 and 27 credits earned in year 2 for HL-SCI 
students, and results are similar for the comparison students (figure 25). 


Figure 25. Regression-adjusted mean number of credits eamed for students in the 
comparison and HL-SCI groups, after one and two yearsof program 
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Notes: Results are from a linear regression model that accommodates for clustering of 
students within colleges (N=5,665 in year 1 and 3,295 in year 2). The model is used to 
calculate the mean number of credits eamed by HL-SCI status, while holding constant all 
other variables. The coefficient on the HL-SCI variable is -0.266 (p=0.499) in year 1 and 
1.289 (p=0.324) in year 2. 


* Coefficient (of estimated intention-to-treat impact) is statistically significant at the 95 
percent confidence level using a two-tailed test. 


Credit accumulation tends to differ by program category, particularly in the 
comparison group. Comparison students in health/medical programs tend to have 
more credits earned after one year of program enrollment than students in all 
programs, while comparison students in science programs tend to have fewer credits 
earned after both one and two years of program enrollment (table 17). There are 
fewer differences in credit accumulation outcomes in the intervention group. HL-SCI 
students in science programs earn an average of 22.1 credits after two years, 
compared with 27.4 credits for HL-SCI students in all programs, holding other 
factors constant. This difference of 5.3 credits is equivalent to between one and two 
courses, aS most courses carry 3 credits. 
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Table 17. Regression-adjusted mean number of creditseamed by program category, 
afterone and two years of program enrollment 


Comparison group Intervention group 
Credits eamed: One year 
95% 95% 
confidence confidence 
N Margin interval N Margin interval 
Health/medical 3,079 16.2) 15.7 16.6 865 15.0 143 15.7 
Science 477 12.8 11.8 13.7 248 15.3 141 16.5 
Data/IT 756 14.4 13.6 15.1 240 13.8. 12.3 14.4 
All 4,312 15.0 14.7 15.5 1,353 14.7. 141 15.3 
Credits eamed: Two years 
95% 95% 
confidence confidence 
N Margin interval N Margin interval 
Health/medical 1,810 28.3 27.0 29.6 481 28.5 26.7 30.3 
Science 318 18.4 15.5 21.4 106 28.5 25.1 32.0 
Data/IT 462 24.8 22.8 26.9 118 22.1 19.2 249 
All 2,590 26.2 25.0 27.4 705 27.4 256 288 


Notes: Results are from a linear regression model that accommodates for clustering of 
students within colleges (N=5,665 in year 1 and 3,295 in year 2). The model is used to 
calculate the mean number of credits eamed by HL-SCI status, while holding constant all 
other variables 


Effectiveness of HL-SCI components 


After assessing the overall impact of the HL-SCI grant, we examined whether there 
are any differences in outcomes for HL-SCI students based on whether they 
participated in grant-related components for PLAs, booster modules, and online and 
hybrid courses. 


Prior leaming assessments 


To evaluate the impact of PLAs, we use the same regression models as for the overall 
impact of program effectiveness, but we add a dichotomous variable equal to one for 
HL-SCI participants who received any PLA credit and zero for HL-SCI participants 
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who did not receive any PLA credit.'* For students with at least one year of follow-up 
data, we also examine whether the results differ if we use a categorical variable that 
quantifies the number of PLA credits received. This allows us to check if the 
relationship between PLA credits is not linear. For example, getting credit for a few 
courses helps students complete programs faster, but getting many PLA credits 
means students are unprepared for higher-level courses in the program. Data on PLA 
credits were not available for comparison students enrolled prior to HL-SCI. 


The impact of PLAs on outcomes for HL-SCI participants appears to be mixed (table 
18). On the negative side, participants with PLAs were less likely to persist in college 
after one year than students without PLAs (58.69 percent versus 69.88 percent, a 
difference of 11.19 percentage points). However, the two groups had _ similar 
outcomes for college persistence after two years of enrollment in the program. On 
the positive side, participants with PLAs were more likely to complete credentials 
after one and two years of program enrollment, particularly among students in 
associate’s degree programs. The predicted probability of completing an associate’s 
degree was 5.20 percentage points higher for participants with PLAs after one year, 
and 33.42 percentage points higher for participants with PLAs after two years. There 
were no statistically significant differences in the total number of credits earned 
through coursework during the study period among participants with and without 
PLAs. 


\“ This analysis does not include comparison students because data on PLA credits were 
available only for HL-SCI participants. 
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Table 18. Student outcomes for HL-SCI participants with and without PLA credits, after 
one and two yearsof program enrollment 


1 Year 2 Years 

Persistence 

Participants with PLAs 58.69% 75.37% 

Participants without PLAs 69.88% 65.65% 

Difference -11.19% * 9.72% 
Credential: any 

Participants with PLAs 5.59% 45.22% 

Participants without PLAs 1.31% 15.28% 

Difference 4.28% * 29.94% * 
Credential: associate's degree 

Participants with PLAs 5.71% 44.46% 

Participants without PLAs 0.51% 11.04% 

Difference 5.20% * 33.42% * 
Credential: certificate 

Participants with PLAs a 40.74% 

Participants without PLAs 2 37.53% 

Difference a 3.21% 
Credits eamed 

Participants with PLAs 14.70 27.63 

Participants without PLAs 14.74 27.44 

Difference -0.04 0.19 


Notes: Results are from a logit model (for dichotomous variables) or linear regression model 
(for continuous variables) that accommodates for clustering of students within colleges. 
The model isused to calculate the probability of each dichotomous variable by PLA status 
(orthe regression-adjusted mean for continuous variables by PLA status), while holding 
constant all other variables. 


* Coefficient (of the PLA variable) is statistically significant at the 95 percent confidence 
level using a two-tailed test. 


@ Fewerthan 10 HL-SCI students completed a certificate in one year, and none had 
received PLA credit. 


Next, we examined whether the impact of PLAs differs depending on the number of 
credits earned through the PLA process (table 19). Results are presented only after 
one year of program enrollment due to the small number of students with PLAs in 
each category in the year 2 follow-up sample. For the college persistence outcome, we 
find that the probability of persistence is lowest for students with 4 to 6 PLA credits 
(55.88 percent) and highest for students with no PLA credits (69.89 percent). 
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Table 19. Student outcomes for HL-SCI participants by number of PLA credits received, 
afterone yearof program enrollment 


Persistence 

No PLA credits 69.89% 

3 or fewer PLA credits 59.03% 

4 to 6 PLA credits 55.88% * 

More than 6PLA credits 66.39% 
Credential: any 

No PLA credits 1.30% 

3 or fewer PLA credits 3.87% ~ 

4 to 6PLA credits 3.50% ~ 

More than 6PLA credits 13.66% * 
Credential: associate's degree 

No PLA credits 0.50% 

3 or fewer PLA credits 4.13% * 

4 to 6PLA credits 5.49% * 

More than 6PLA credits 7.32% * 
Credential: certificate 

No PLA credits a 

3 or fewer PLA credits a 

4 to 6 PLA credits a 

More than 6 PLA credits iz 
Credits eamed 

No PLA credits 14.76 

3 or fewer PLA credits 14.33 

4 to 6 PLA credits 14.29 

More than 6PLA credits 16.68 


Notes: Results are from a logit model (for dichotomous variables) or linear regression model 
(for continuous variables) that accommodates for clustering of students within colleges. 
The model isused to calculate the probability of each dichotomous variable by the 
number of PLA credits received (orthe regression-adjusted mean for continuous variables 
by the numberof PLA credits received), while holding constant all other variables. 

* Coefficient (of the PLA variable) is statistically significant at the 95 percent confidence 
level using a two-tailed test. The reference group is students with no PLA credits. 

~ Coefficient (of the PLA vanable) is statistically significant at the 90 percent confidence 
level using a two-tailed test. The reference group is students with no PLA credits. 

a Fewer than 10 HL-SCI students completed a certificate in one year, and none had 
received PLA credit. 


For the credential completion outcomes, there is some evidence that the probability 
of completing a credential increases as the number of PLA credits increases. For 
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example, the predicted probability of completing an associate’s degree is lowest for 
participants with no PLA credits (0.50 percent), followed by participants with 3 or 
fewer PLA credits (4.13 percent), then participants with 4 to 6 PLA credits (5.49 
percent), and highest for participants with more than 6 PLA credits (7.32 percent). 
These results are as anticipated, since PLA credits should allow students to progress 
through their programs more quickly than if they had to accumulate all of their 
credit hours through new coursework. 


Booster modulesand online and hybnd courses 


Finally, we examine how course outcomes differ for HL-SCI participants who 
participated in booster modules or online and hybrid courses. We compare outcomes 
for an intervention group of HL-SCI participants who have enrolled in a booster or 
online/hybrid course with those of a comparison group consisting of students who 
enrolled in a prior year in the same course without the HL-SCI component. 
Regression models are estimated separately for the booster component and the 
online and hybrid component. 


The first outcome of interest is course completion, which is a dichotomous variable 
for whether the student successfully completed the course with a grade of C- or 
above (which is considered passing by the Consortium colleges). Students with a 
grade of A to C- are coded as one, while students with a grade of D+ to F are coded 
as zero. The second outcome is a continuous variable for course grade on a scale 
from 0 (F) to 4 (A), which corresponds to grade point average (GPA) calculations used 
by the Consortium colleges.’ Course grades are available only for the subset of 
students who complete the course. 


We find that course completion rates are high (above 90 percent) for all students 
regardless of whether the course included any HL-SCI components (figure 26). There 
are no Statistically significant differences for the HL-SCI components in either model, 
which suggests that the booster modules and the online and hybrid course format do 
not influence the likelihood that students will complete courses. 


* Course grades were assigned numeric values as follows: A=4.0, A-=3.7, B+=3.3, B=3.0, B-=2.7, 
C+=2.3, C=2.0, C-=1.7, D+=1.3, D=1.0, D-=0.7, and F=0.0. 
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Figure 26. Predicted probability of course completion for students in courses with and 
without booster modules, and courses with and without online or hybrid 
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Notes: Results are from a logit model that accommodates for clustering of students within 
colleges (N=9,597 forthe booster module analysisand N=/,408 forthe online and hybrid 
analysis). The model isused to calculate the probability of course completion by HL-SCI 
component status, while holding constant all other variables. The coefficient on the 
booster variable is -0.188 (p=0.190) and the coefficient on the online and hybrid variable is 
0.712 (p=0.206). 


* Coefficient (of the HL-SCI component) is statistic ally significant at the 95 percent 
confidence level using a two-tailed test. 


There is some evidence that course grades may be higher for students in online and 
hybrid courses than for students in the same course in traditional in-person format 
(figure 27). The regression-adjusted mean course grade is 3.20 for online and hybrid 
courses, compared with 3.04 for traditional courses, a difference of 0.16 (p=0.05). 
However, differences in unobserved characteristics between the two groups may 
contribute to differences in course grades. For example, students who choose to take 
online and hybrid courses may have greater internal motivation, which may also 
influence course grades. It is also possible that the content of the courses may have 
changed before and after the HL-SCI component for online and hybrid courses, which 
may also contribute to differences in course grades. Booster modules did not 
contribute to significant differences— course grades were similar (around 2.9, which 
is almost a B) for students in the same courses with and without booster modules. 
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Figure 27. Regression-adjusted mean course grade for students in courses with and 
without booster modules, and courses with and without online or hybrid 


format 
No booster m Booster Not online m Online 
4.00 - 4.00 , 
aaa ee | 3.20% 
3 2.97 3.04 
© 3.00 + 3.00 - 
oD 
3 
5 2.50 + 2.50 - 
° 
15) 
c 
§ 2.00 - 2.00 + 
= 
1.50 + 1.50 - 
1.00 1.00 
Course grade by Course grade by 
booster status online/hybrid status 


Notes: Results are from a linear regression model that accommodates for clustering of 
students within colleges (N=9,467 for the booster module analysis and N=/,362 for the 
online and hybrid analysis). The model is used to calculate the mean course grade by HL- 
SCI component status, while holding constant all other variables. The coefficient on the 
booster variable is -0.027 (p=0.451) and the coefficient on the online and hybrid variable is 
0.167 (p=0.051). 


* Coefficient (of the HL-SCI component) is statistic ally significant at the 95 percent 
confidence level using a two-tailed test. 
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Summary 


This report provides the final results from an independent, third-party evaluation of 
the Connecticut HL-SCI. The implementation evaluation assesses actual performance 
relative to grant deliverable goals, and the qualitative analysis provides feedback on 
how the design and implementation of HL-SCI creates both opportunities for and 
barriers to student success. The quantitative evaluation of the grant’s impact 
assesses actual performance relative to goals for student outcomes related to 
enrollment and completion of HL-SCI programs of study and continuing education 
and employment of program completers. It also employs a quasi-experimental design 
to determine the impact of the HL-SCI on student outcomes of college persistence, 
credential completion, and credit accumulation. The results for both the 
implementation and impact analyses are summarized below. 


Implementation evaluation 


Actual performance relative to grant deliverable 
goals 


The HL-SCI grant exceeded all 17 deliverable goals, which were designed to measure 
progress toward each component of the HL-SCI grant: the new or revised certificate 
or degree programs, booster modules in math and science, online and hybrid 
courses, PLA, and internships and job placements. This indicates that the 
Consortium greatly expanded access to HL-SCI programs of study, the types of 
programs offered, and the availability of academic supports. 


Program enrollment and recruitment 
recommendations 


Increase efforts to recruit students through Consortium activities. Even though the 
Consortium exceeded its goals for the number of participants, relatively few study 
respondents reported learning about their programs through targeted grant 
activities. Most HL-SCI students learned about their programs independently, either 
through an online search or word-of-mouth from friends or family. However, those 
students who did learn about their program through contact with a faculty or staff 
member at their college found this input beneficial. A third of survey respondents 


66 


CNA 


indicated that staff member or professors at their colleges were “extremely 
influential” in their decisions to enroll. The colleges might consider how to target 
recruitment activities so that additional students are recruited through Consortium 
activities. Data on why students chose their programs may be helpful to these 
efforts. For example, most students enrolled in HL-SCI programs of study because 
they were interested in the subjects they were studying and/or wanted to pursue a 
new job or career path. Convenience and affordability were also important factors in 
their enrollment decisions. 


Have college staff members guide students in selecting a program of study. 
Advising is particularly important for prospective students who are considering 
multiple programs. Focus group and interview participants and survey respondents 
who received guidance from college staff members in selecting a program prior to 
enrolling found it helpful. Colleges should also ensure that advising staff have 
adequate training to help students understand their options among the programs of 
study and the career pathways for each. 


Booster modules recommendations 


Maintain and expand student access to booster modules. Most of the students who 
provided feedback through interviews and focus groups had not participated in 
booster modules and were unfamiliar with them. The colleges might consider 
expanding access to booster modules, given that self-reported data indicate that they 
support student success. With few exceptions, those who had taken booster modules 
found them useful because they provided another method through which to learn 
course material. Students who completed the Survey Monkey questionnaire on 
booster modules also offered positive feedback overall. They found the boosters 
appropriately challenging and easy to follow. The majority would recommend the 
booster modules they completed to other students. 


Online and hybrid course recommendations 


Improve student engagement and interaction with faculty in online and hybrid 
courses. Most focus group and interview participants preferred in-person courses to 
the online and hybrid formats because the traditional courses provided the 
opportunity for more interaction between students and professors. Students could 
also have their questions answered immediately. However, some faculty members 
found ways to ameliorate these challenges. For example, a number of students said 
that their instructors created in-person lab sessions or offered office hours to 
increase interaction between faculty and students. 


Continue to offer online and hybrid courses to ensure flexibility for students. 
Although many students expressed a preference for in-person classes, some students 
appreciated online and hybrid courses because they were convenient and they could 
complete content at their own pace. Sixty-four percent of student survey respondents 
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had completed at least one online or hybrid course. These students also found them 
convenient and easy to access. Nearly two-thirds would recommend online or hybrid 
courses to others. 


Further investigate why students perceive that they learn more in in-person 
courses. Only half of the students believed that they learned as much in their online 
or hybrid courses as they would in a traditional course. The Consortium should 
collect additional student feedback on why they believe that they learn more in in- 
person courses than online and hybrid courses. 


Prior leaming assessment recommendations 


Ensure that students who might benefit from prior learning assessments take 
advantage of the opportunity. Generally, interview and focus group participants 
were aware of the prior learning assessment process, but few students had applied 
for a review of their prior learning experiences. It is unclear whether these students 
had the prior experience and education to benefit from the prior learning process. 
The Consortium should ensure that the criteria for the prior learning assessment 
process are clear and widely disseminated. Most students who applied previously 
earned credit toward their degrees did so through the more traditional process of 
transferring academic credit from another postsecondary institution. 


For the most part, students found the process of transferring credits straightforward 
and fair. Similarly, the majority of survey respondents indicated that the process for 
receiving prior learning credit was easy to understand and that they received the 
right amount of credit based on their prior knowledge and experience. Nearly 70 
percent believed they would complete their programs in a shorter time frame as a 
result of receiving credit. 


Ensure that the process for awarding credit is straightforward and transparent. 
Some interview and focus group respondents experienced administrative challenges 
or would have liked more transparency in how their college awarded prior learning 
or transfer credit. For example, colleges could create a list of common experiences 
that can be transferred with the corresponding credits that could be received or a 
guide to CLEP exams with suggestions on how to study for them. 


Employment and placement service 
recommendations 


If possible, ensure that clinical hours are flexible so that students can meet other 
obligations. Most students who participated in internship experiences as part of 
their program had completed clinical practicums. Students liked that clinical 
experiences were hands-on and allowed them to apply what they had learned in the 
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classroom. Some students would like more flexible scheduling for clinical hours, 
which would allow them to simultaneously hold full- or part-time jobs. 


Expand access to college and career guidance for continuing students and 
graduates. Focus group and interview participants who received career guidance 
from their colleges generally found them helpful. Additionally, the majority of survey 
respondents who received specific career guidance or placement services found them 
very or somewhat helpful. Students especially appreciated employment feedback 
from faculty members who had worked in their fields. Some participants expressed 
the need for additional career guidance and placement services at their colleges. 
These opportunities should be sustained and expanded. 


Recommendations from general student feedback 
from focus groups and interviews 


Ensure that course content continues to be applied and hands-on. Overall, students 
provided very positive feedback on their programs and professors. They liked that 
courses were offered in a hands-on format that allowed them to apply their skills in 
both laboratory and real-world settings. 


Maintain the quality of professors and their pedagogical approach. Students 
praised their instructors, indicating that they were engaging and accessible. They 
liked that most of their instructors had previous experience as professionals in their 
field, which allowed them to connect lectures and laboratory sessions to on-the-job 
situations. 


Continue to provide additional support for veterans. Veteran students provided 
positive feedback about the targeted supports they received on their campuses, such 
as having a designated veterans’ coordinator on campus. Additionally, the grant 
allowed the Consortium to develop booster modules to help veteran students with 
their transition to college and provide college staff with specialized information on 
advising veterans. 


Ensure that students have access to courses and facilities at a time that meets 
their schedules. The biggest challenge for many students was balancing coursework 
with family and work responsibilities. A few students suggested offering courses at 
additional times that would be convenient to their schedules. Another challenge for 
some students was perceived limited access to facilities that supported their 
academic progress, such as laboratories and therapeutic gymnasiums. 
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Impact evaluation 


Actual performance relative to student outcome 
goals 


The HL-SCI grant exceeded seven of the nine student outcome goals, which were 
designed to measure progress toward student enrollment and completion of HL-SCI 
programs of study and continuing education and employment of program 
completers. The two goals that were not met were related to initial employment and 
sustained employment after study completion for students who were not employed 
at initial program enrollment. However, there was considerable variation in these 
outcomes by program type and college. In addition, many HL-SCI participants were 
still enrolled in their programs of study at the end of the grant period so there was 
insufficient follow-up time to assess these outcomes for the majority of participants. 


Overall impact of the HL-SCI grant 


We examined the overall impact of the HL-SCI grant on student outcomes, including 
college persistence, credential completion, and credit accumulation. Outcomes for 
HL-SCI students were compared with those of a matched comparison group of 
students enrolled in the same or similar programs of study prior to the new or 
revised HL-SCI programs of study. Overall, we found that results were similar on all 
outcomes for the HL-SCI students and the matched comparison students one year 
and two years after initial program enrollment. However, this evaluation included 
only students who entered a HL-SCI program of study in the first two years of the 
grant and tracks outcomes for one to two years, so it is possible that it may be too 
early to detect an impact of the HL-SCI grant. 


Although there are several differences in outcomes by program category in the 
comparison group, HL-SCI students tend to perform similarly regardless of program 
category. The one exception is that HL-SCI students in science programs tend to 
complete 5 to 6 fewer credits after two years of program enrollment than HL-SCI 
students in all programs. One reason might be that courses required for science 
programs may be more rigorous than those required for some other HL-SCI programs 
of study. For example, students may have difficulty taking upper-level biology and 
chemistry courses in the same semester. 


Effectiveness of HL-SCI components 


We also conducted separate effectiveness analyses for three components of HL-SCI: 
PLAs, booster modules, and online and hybrid courses. Results for PLAs were mixed. 
On the positive side, students who received PLA credit were more likely to complete 
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a credential than participants without PLA credits. PLA credits should allow students 
to progress through their programs more quickly than if they had to accumulate all 
of their credit hours through new coursework, so this may contribute to higher 
credential completion rates, particularly given the short follow-up period of only one 
to two years. On the negative side, participants who received PLA credits were less 
likely to persist after the first year than participants without PLA credits. The 
reasons for the negative impact of PLAs on first-year persistence are uncertain, but it 
could be that students may not possess the background they need to master material 
in more advanced courses if PLA credits are granted too freely. 


In addition, we compared course outcomes (course completion and course grade) for 
an intervention group of HL-SCI participants who enrolled in a booster or online and 
hybrid course with those of a comparison group consisting of students who enrolled 
in a prior year in the same course without the HL-SCI component. Course completion 
rates were high at over 90 percent for all students, regardless of course format or 
component. Students in online and hybrid courses had slightly higher course grades 
than students in a traditional in-person version of the same course. Differences in 
unobserved characteristics between the two groups may contribute to differences in 
course grades. For example, students who choose to take online and hybrid courses 
may have greater internal motivation, which may also influence course grades. It is 
also possible that the content of the courses may have changed before and after the 
HL-SCI component for online and hybrid courses, which may also contribute 
differences in course grades. There was no difference in course grades for students 
in courses with and without booster modules. Although it is uncertain why boosters 
did not have a measurable impact on course outcomes, students who participated in 
the implementation evaluation reported that boosters generally increased their level 
of understanding of a subject, so colleges may want to continue to offer boosters. 
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Technical Appendix 


This technical appendix provides additional detail on the data, methods, and findings 
from the impact evaluation. It begins by describing the data sources and the key 
variables constructed from each. Next, it describes the process for identifying the 
analytic sample and assigning students to intervention or comparison conditions. 
Third, it details the matching procedures used to match students in the intervention 
group to similar students in the comparison group. Fourth, it defines the statistical 
models used to estimate the impact of the intervention. Finally, it provides detailed 
results from the regression models used to estimate the impact of the intervention. 


Data sources and key vanables 


The primary source of data for the impact evaluation consisted of student-level 
records provided by each of the college’s institutional research departments. The 
files received included: 


e FO2: Course transcript records including course number, credit hours 
attempted and earned for each course, and course grades. 


e O05: Student census data including demographic characteristics (such as race 
and gender) and program of study. 


e F22 AC: Academic records including semesters of enrollment, credit hours 
attempted and earned for each semester, and GPA in each semester. 


e F22 TS: Accuplacer test score records. 
e H08: Certificate and degree completion records. 


Each file contained a unique student identifier, which we used to merge together all 
of the files for each college. HL-SCI curriculum innovation coordinators maintained 
their own files in Excel or Access to track HL-SCI participants, and these files 
contained many of the same variables. However, we used the data from the 
institutional research departments instead to ensure that variables were being 
collected and coded consistently across colleges, as well as between the intervention 
and comparison group students. 
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The institutional research data was used to construct the outcome measures for the 
overall impact of the HL-SCI grant, which were selected because they are 
manifestations of the goals of the HL-SCI grant. Table Al describes each of the 
outcome variables for the impact evaluation, including the variable type (continuous 
or dichotomous), the data source, and the timing of the variable. 


Table Al. Summary of outcome vanables forthe overall impact of the HL-SCI grant, 


by research domain. 


Data source 


Timing of 
variable 


College 
Persistence 


Whether the student 
persisted in college 1 
or2 years after initial 
program enrollment 


Dichotomous (O=no 
longer enrolled, 1=still 
enrolled or 
completed the 
program) 


F22 AC and H08 


College enrollment in 
the same semester 1 
yearand 2 years 
after program entry 


Research domain 


Credental 
completion 


Whether the student 
completed a 
credential after 1 or 2 
years of program 
enrollment 


Dichotomous (O=did 
not complete a 
credential within the 
follow-up period, 
1=completed a 
credential) 


Credentialseamed in 
any semester within 1 
or2 years after 
program entry 


Credit 
accumulation 
Total number of 
credits eamed after 1 
or2 years of program 
enrollment 


Continuous (number 
of credits) 


Creditseamed by the 
end of the last 
semester 1 yearand 
2 years after program 
entry 


The analyses of the effectiveness of the booster module and online and hybrid HL- 
SCI components used data from the course transcript records to compare outcomes 
for students enrolled in the same courses with and without the HL-SCI components. 
The sample includes the students from the overall impact analyses enrolled in one of 
these courses, as well as some additional students in these courses who had less 
than one year of follow-up data. A summary of the outcome variables for these 
analyses is provided in table A2. 
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Table A2. Summary of outcome vanables forthe effectiveness of booster modules 
and online and hybnd courses, by research domain 


Research domain 
Course completion Course grade 


Whether the student successfully Grade eamed in the course 
completed the course witha 
grade of C- orabove 


Variable type Dichotomous (O=course grade of Continuous (scale of 0 forF to 4 
D+to F, l=course grade ofAtoC-) forA) 


Data source 


Timing of End of course End of course 
variable 


The data for the analysis of the effectiveness of HL-SCI components also includes 
lists provided by the HL-SCI coordinators of courses that contain booster modules at 
each college by semester, as well as lists of online and hybrid courses offered by 
each college by semester. In addition, the HL-SCI staff provided data on the number 
of credits HL-SCI participants earn through PLAs. 


Procedures used to identify students in the 
sample 


One of the challenges to the impact evaluation was identifying which students 
should be categorized in the intervention group versus the comparison group 
because many students had noncontinuous patterns of enrollment both before and 
after the implementation of the HL-SCI grant. If we restricted the intervention group 
to students who were enrolled only after the HL-SCI grant and the comparison group 
to students who were enrolled only prior to the HL-SCI grant, we would lose a large 
portion of the students served by the grant, which would limit the generalizability of 
the findings. Another challenge to identifying the treatment and comparison groups 
is that new and revised HL-SCI programs of study were rolled out over several 
semesters, so the intervention did not always begin along with the grant initiation in 
spring 2013. 


To address these challenges, we used the new or revised program start dates to 
define when the intervention began for each program of study. We then categorized 
three types of students who belonged to the intervention group: 


e New students: Students who earned their first credit after their program of 
study was added or revised under the HL-SCI grant were coded as new 
students in the intervention group. 
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e Continuing students: If students took a course in any semester (spring, 
summer, fall, winter) during the academic year prior to the new or revised 
program start dates AND after the start of the HL-SCI grant, they are defined 
as continuing students. Even though their specific program of study had not 
yet been added or revised, they may have benefitted from other activities or 
services provided under the grant as a whole. For example, they may have 
taken prerequisite courses common to other HL-SCI programs of study that 
had already been updated to include booster modules or online and hybrid 
course formats. For this reason, we also code these students as belonging to 
the intervention group, although we conduct sensitivity analyses to see if the 
results change if these students are excluded from the sample. 


e Returning students: Students who were not enrolled in the year prior to the 
new or revised HL-SCI programs of study (not continuing students), but 
earned their first credit more than one year prior to this time are categorized 
as returning students. A discussion with an HL-SCI staff member indicated 
that Consortium colleges typically consider students to have dropped out if 
they have not taken classes in any semester for more than a year. The HL-SCI 
grant serves a large number of students who are returning to complete their 
credential after previously dropping out, or beginning a new credential 
program in a different field of study. These students are coded in the 
intervention group in the first semester that they begin taking courses once 
they return to college. In order to control for length of time that returning 
students have been out of college, we include a continuous variable in the 
regression models for the number of semesters between the student’s first 
term and the first term in the study period. 


The comparison group consists of students who were enrolled in the same or similar 
programs of study at Consortium colleges prior to the start of the grant. The 
students in the comparison group entered their program of study between fall 2009 
and winter 2011. This allows up to one full year of follow-up (until winter 2012) 
before the HL-SCI grant began implementation in spring 2013. We only track 
outcomes two years after initial program enrollment if there is sufficient follow-up 
time prior to implementation of HL-SCI. For example, comparison students who enter 
their program of study in fall 2009 can be tracked through fall 2011 so they have two 
years of follow-up data. However, students who enter their programs of study in fall 
2011 can only be tracked for one year through fall 2012 since the HL-SCI grant would 
have started during their second year of program enrollment. 


Table A3 provides a summary of students in the intervention and comparison groups 
by student status. The full sample consists of 1,836 intervention students and 5,464 
comparison students for a total of 7,300 students. This includes some students with 
less than one full year of follow-up, who are included only in analyses of the 
effectiveness of booster modules and online and hybrid course HL-SCI components. 
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Table A3. Summary of students in the intervention and comparison groupsin the full 


sample (N=/,300), by student status 


Intervention group (N=1,836) 


Student status 


New student 


Continuing 
student 


Retuming 
student 


Description 


Student eams first credit 
afternew/revised program 
started. 


Student was enrolled in any 
semester in the year prior to 
the program being 
new/revised and continued 
on to a new/revised 
program. 


Student was not enrolled in 
the year priorto the 
program being new/revised, 
but had their first credit at 
an earliertime. 


Comparison group (N=5,464) 


Student status 


New student 


Continuing 
student 


Retuming 
student 


Description 


Student eams first credit 
during comparison period 
(fall 2009 to winter 2011). 


Student was enrolled in any 
semester in the year prior to 
the first course in an HL-SCI 

comparison program. 


Student was enrolled during 
comparison penod, but first 
credit wasatan eariertime. 


Analytic procedures 


Include in all analyses. 


Include in primary analyses 
(in the body of the report). 
Conduct sensitivity 
analyses to see if the 
results change if they are 
excluded from the sample. 


Include in all analyses. 
Control forthe number of 
semesters between first 
term everand first term in 
the study penod. 


Analytic procedures 


Include in all analyses. 


Include in primary analyses 
(in the body of the report). 
Conduct sensitivity 
analyses to see if the 
results change if they are 
excluded from the sample. 
Include in all analyses. N=1,859 
Control for the number of 

semesters between first 

term everand first term in 

the study period. 


The number of students in the intervention group for the evaluation differs from the 
number of participants in each semester in HL-SCI reports to the U.S. Department of 
Labor. Consortium colleges differed in how they classified students’ term of HL-SCI 
entry. For example, some colleges coded students as HL-SCI participants once they 
had enrolled in the college and indicated their intent to major in an HL-SCI program 
of study, whereas other colleges coded students as HL-SCI participants after they had 
completed all of their prerequisites and been accepted into an HL-SCI program of 
study. This means that students classified as HL-SCI participants by grant staff may 
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have completed different amounts of progress toward their program of study 
depending on the college they attended. The procedures used in the impact 
evaluation to assign students to the intervention group ensure that all students are 
coded as HL-SCI participants at the same point of program entry. 


Matc hing procedures 


After identifying the students in the full sample, we used matching methods to 
create an analytic sample of intervention and comparison students who are similar 
on observed characteristics. Matching methods reduce the selection bias that is 
associated with nonrandom assignment into intervention and comparison groups by 
forming a sample of comparison students that is similar to the group of intervention 
students. 


We implemented coarsened exact matching (CEM) to obtain a matched sample of 
treatment and comparison students. We used CEM as opposed to exact matching 
because some of the matching variables are continuous (for example, program 
duration), which can make it hard to obtain matches. CEM temporarily coarsens 
continuous variables into categorical variables during the matching process, in which 
the category bin size is determined in a way that maintains balance across the 
treatment and control groups while also maximizing the sample size (lacus, King, & 
Porro, 2011). Then the uncoarsened, matched data can be used in a statistical model 
by weighting observations based on the size of their strata. We also avoided 
propensity score matching because the standard errors from the treatment model 
are biased as the treatment model includes the estimated propensity score as a 
regressor. Additionally, propensity score matching “increases imbalance, inefficiency, 
model dependence, and bias,” which can be avoided by using a randomized block 
design such as CEM (King & Nielsen, 2016, p. 1). 


Our matching variables are summarized in table A4. We used an exact match for the 
student status variable to ensure that the comparison and treatment groups are 
comparable in terms of prior college experiences. Additionally, we required an exact 
match by program category so that both groups are similar in terms of program 
content and occupational outlook.'* Lastly, we used a coarsened exact match to 
identify students in programs of similar duration. This is important because it allows 
credential completion rates to be compared among students in programs that take a 
similar amount of time to complete. 


© Programs of study were categorized based on input from all of the HL-SCI curriculum 


innovation coordinators, who are familiar with the programs offered at each college. 
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Table A4. Summary of matching variables 


Characteristicsto Variables included Type of match 

match 

Student status Whether the student isa new, continuing, or Exactmatch 
retuming student. 


Program category Whether the student's program of study is Exact match 
classified as health/medical, science, or 
data/IT. 

Program duration Numberof credit hours required for Coarsened exact 
credential completion, ranging from 8 to 69. match 


Observations are dropped if the stratum does not include at least one treated 
student and one comparison student. In our analysis, 1,364 of the 1,366 HL-SCI 
students (99.9 percent) and 5,428 of the 5,461 of the potential comparison students 
(99.4 percent) are matched. The observations remaining in the analytic sample are 
weighted according to the size of their strata. 


Baseline equivalence of the analytic sample 


After obtaining the matched sample, we conducted a preliminary analysis to 
establish whether the intervention and matched comparison groups were similar at 
baseline (prior to program entry). We compared the means of the intervention and 
comparison groups on the following characteristics: 


e Student demographic characteristics: percent female, percent Black, percent 
Hispanic, and age. 


e Test scores: scale scores on the Accuplacer assessment in algebra, arithmetic, 
reading, and sentence skills. 


e Student characteristics: number of terms enrolled prior to study entry, and 
percent enrolled full-time. 


e College attended (Capital, Gateway, Manchester, Middlesex, or Norwalk 
Community College). 


To examine whether any differences exist between the HL-SCI students and the 
matched-comparison students, we examine the absolute standardized bias on each of 
the baseline characteristics. This is calculated using the equation: 


Xi — xX c 
Absolute standardized bias=|-—————_ 


ic 
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where X; is the mean of the intervention group, x is the mean of the comparison 
group, and o,,is the pooled standard deviation of the intervention and comparison 


groups. The absolute standardized bias is a diagnostic of the balance between the 
intervention and comparison groups on the variables of interest. 


Stuart (2007) recommends that absolute standardized bias values greater than 0.50 
are “particularly problematic,” and should ideally be less than 0.25. Tables A5 and 
A6 show the absolute standardized bias and level of bias for each of the baseline 
variables among students with at least one year and two years of follow-up data, 
respectively. After matching, no variables have an absolute standardized bias greater 
than 0.50, and most are also less than 0.25. 


Among the variables with moderate bias, the intervention group is slightly younger 
(average age of 28 years versus 32 years for the comparison group in the one-year 
follow-up sample) and the distribution of colleges differs somewhat because some 
campuses expanded more rapidly than others after the HL-SCI grant began. 


To further control for any remaining differences between the intervention and 
comparison groups, we include covariates in the regression models for all baseline 
characteristics except test scores. Many students are advised to take the Accuplacer 
upon college entry to help determine their level of placement into their first math or 
English course, but testing is not required. The majority of students in our sample do 
not have any Accuplacer scores, so too many students would be dropped from the 
sample due to missing data if test scores were included as covariates in the impact 
models. Although bias levels are low among students with test scores, there may be 
unobserved differences in academic ability among students who did not take the 
Accuplacer exam. 
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Table A5. Baseline equivalence for students in the matched sample with at least one year of follow-up data 


Comparison group Intervention group 
N Mean N Mean Difference Pooled Std. Bias BiasLevel 
Std. 
Dev. 
Student demographics 
% Female 5428 0.69 1362 0.63 0.06 0.48 0.12 Low 
% Black 5428 0.20 1364 0.15 0.05 0.39 0.14 Low 
% Hispanic 5428 0.17 1364 0.16 0.01 0.38 0.03 Low 
Age 4988 32.10 1361 28.39 3.71 10.90 0.34 Moderate 
Test scores 
Algebra score on Accuplacer 2097 49.88 650 54.87 -4,99 25.64 -0.19 Low 
Arithmetic score on Accuplacer 1132 43.90 271 48.03 -4,13 24.70 -0.17 Low 
Reading score on Accuplacer 1955 71.25 562 73.81 -2.56 23.47 -0.11 Low 
Sentence skillsscore on Accuplacer 1883 79.60 530 82.39 -2.79 21.35 -0.13 Low 


Student characteristics 


% New students 5428 0.50 1364 0.50 0.00 0.50 0.00 Low 
% Continuing students 5428 0.46 1364 0.46 0.00 0.50 0.00 Low 
%Retuming students 5428 0.05 1364 0.05 0.00 0.21 0.00 Low 
Number of terms prior to study entry 5428 -3.15 1364 -3.74 0.59 11.30 0.05 Low 
% Enrolled full-time 5305 0.28 1357 0.37 -0.09 0.46 -0.20 Low 
Program characteristics 

% Health/medical program 5428 0.64 1364 0.64 0.00 0.48 0.00 Low 
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Comparison grou Intervention group 
N Mean N Mean Pooled Std. Bias Bias Level 
Std. 
Dev. 
% Science program 0.18 0.18 . 0.39 0.00 Low 
% Data/ITprogram 0.18 0.18 : 0.38 0.00 


Program duration (number of terms) 60.85 60.66 11.14 0.02 


College attended 

Capital Community College Moderate 
Gateway Community College Low 
Manchester Community College Moderate 
Middlesex Community College Low 
Norwalk Community College Moderate 


Table A6. Baseline equivalence for students in the matched sample with at least two years of follow-up data 


Comparison group Intervention group 
N Mean N Mean Difference Pooled Bias Level 
Std. 
Dev. 
Student demographics 
% Female 0.47 Low 


% Black 0.39 Low 
% Hispanic 0.37 Low 
Age : 10.89 Moderate 


Test scores 
Algebra score on Accuplacer 
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Comparison grou Intervention group 


Mean N Mean Difference Pooled Std. Bias Bias Level 
Std. 
Dev. 
Arithmetic score on Accuplacer 44.46 139 45.14 -0.68 24.75 -0.03 


Reading score on Accuplacer 70.60 322 73.40 -2.80 23.58 -0.12 
Sentence skillsscore on Accuplacer 79.59 298 83.86 -4.27 21.03 -0.20 


Student characteristics 

% New students 

% Continuing students 

% Retuming students 

Number of terms prior to study entry 
% Enrolled full-time 


Program characteristics 

% Health/medical program 

% Science program 

% Data/ITprogram 

Program duration (number of terms) 


College attended 

Capital Community College Moderate 
Gateway Community College Low 
Manchester Community College High 
Middlesex Community College Low 
Norwalk Community College Moderate 
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Summary of sample selection 


Figure Al summarizes how the analytic samples for the overall impact of the HL-SCI 
grant relate back to the full sample, and the number of students who were lost from 
the full sample for various reasons. First, students were lost from the full sample if 
they had less than one year of follow-up data. Whereas almost all comparison 
students could be tracked for at least one full year, about one-quarter of students in 
the intervention group entered their program of study after fall 2014, so they could 
not be followed for a full year by the end of the data collection period in fall 2015. 
Second, students were dropped from the full sample if there were no students in 
programs of similar duration or category in the other treatment status group. Less 
than 1 percent of students in both the intervention and comparison groups were 
unable to be matched. Third, students were dropped from the sample if they were 
missing baseline data on student demographics, student characteristics, or program 
characteristics These baseline variables were included as covariates in the regression 
models to further control for any differences between the intervention and 
comparison groups. Lastly, some students were lost from the full sample because 
they were missing outcome data. Only about 1 percent of students were missing 
school enrollment records, which were used to create the outcome variable college 
persistence. None of the students were missing data on credential completion 
because it was assumed that a student had not completed a credential if they did not 
have a credential record. For the credits accumulated outcome, data are missing in 
the comparison group for continuing students who started taking courses in the year 
prior to fall 2009 because this was the first semester for which transcript records 
were available. 
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Figure Al. Summary of the samples forthe overall impact of the HL-SCI grant after 
one year of follow-up 


Full sample 
Intervention: 1,836 
Comparison: 5,464 


Sample with at least 1 year follow-up Lessthan 1 year follow-up 
Intervention: 1,366 Intervention: 460 
Companion: 5,461 Comparison: 3 


Matched sample Unmatched 
Intervention: 1,364 Intervention: 2 
Companion: 5,428 Companion: 33 


Matched sample with baseline data Missing baseline data 
Intervention: 1,353 Intervention: 11 
Companison: 4,865 Comparison: 563 


Analytic samples: Missing outcome data 
Persistence outcome: Persistence outcome: 
Intervention: 1,337 Intervention: 16 

Companison: 4,860 Comparison: 5 
Any credential outcome: Any credential outcome: 
Intervention: 1,353 Intervention: 0 
Companison: 4,865 Comparison: 0 
Credits eamed outcome: Credits eamed outcome: 
Intervention: 1,353 Intervention: 0 
Companison: 4,312 Comparison: 553 


Statistic al models for impact analysis 


After establishing the matched sample, we used weighted regression analysis to 
obtain the treatment effect for each of the outcomes of interest. Weights were used 
to take into account the matching process, so that comparison units within each 
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stratum are weighted to equal the number of intervention units in that stratum. The 
credit accumulation outcome is a continuous variable and was estimated using 
ordinary least squares. For the binary outcomes, we used a logit model specification 
with maximum likelihood estimation. The effect of the HL-SCI grant for each 
outcome is then obtained by estimating the following model: 


y; = a+ B(intervention status;) + (baseline covariates;)@ + €; 
Here, the outcome (y) is a function of the following: 


e #: The intervention status, which is equal to one if the student is in a HL-SCI 
program of study under the TAACCCT grant and equal to zero if the student is 
in a comparison program. 


e @: A vector of covariates for baseline characteristics representing student 
demographics, student characteristics, program characteristics and college 
attended (as shown in tables A5 and A6). 


e ¢: Arandom component. 


We clustered the standard errors by college attended. We then obtained the average 
treatment effect on the treated by calculating the average derivative for the 
intervention status indicator. For ease of interpretation, we also report the predicted 
probabilities or regression-adjusted mean for each outcome in the intervention and 
comparison groups. We compared the results for two hypothetical populations—one 
with all HL-SCI students and one with all comparison students—that have the same 
values on all other variables in the model. This shows how outcomes for the two 
groups would differ if all other factors were held constant. 


Full results from the estimates of the impact of 
the intervention 


Table A7 provides detailed regression results for the results presented in the body of 
the report about the overall impact of the HL-SCI grant on student outcomes. Table 
A8 shows the same results for the sensitivity analyses that exclude continuing 
students. 
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Table A7. Full results from the estimates of the overall impact of the HL-SCI grant on 
student outcomes (all students) 


Years of exposure 
Comparison group 
N of students 
Treatment group N 
of students 
Comparison group 
standard deviation 
Treatment group 
standard deviation 
Comparison group 
mean 
Impact estimate 
Impact standard 
enor 


College persistence 

1 4,860 1,337 0.456 0.466 

2 3,134 702 0.476 0.472 
Credential completion: any credential 

1 5,428 1,364 0.104 0.129 

2 3,592 709 0.353 0.375 
Credential completion: associate's degree 

1 4,420 1,214 0.080 0.094 

2 2,838 644 0.308 0.343 
Credential completion: certificate 

1 445 133 0.155 0.219 

2 279 59 0.494 0.484 
Credits accumulated 

1 4,312 1,353 9.067 9.108 14.009 

2 2,590 705 16.657 16.331 23.227 


Notes: Results are from a linear regression model (for continuous outcomes) or logit model 
(for dichotomous outcomes) that accommodates for clustering of students within colleges. 


* Coefficient (of estimated intention-to-treat impact) is statistically significant at the 95 
percent confidence level using a two-tailed test. 
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Table A8. Full results from the estimates of the overall impact of the HL-SCI grant on 
student outcomes (sensitivity analysis excluding continuing students) 


N of students 
of students 


Years of exposure 
Companison group 
Treatment group N 
Comparison group 
standard deviation 
Treatment group 
standard deviation 
Comparison group 
mean 
Impact estimate 
Impact standard 
enor 


College persistence 


1 4,311 712 0.495 0.487 0.573 0.122 0.102 


2 2,589 187 0.499 0.501 0.535 -0.052 0.181 
Credential completion: any credential 


1 4,316 728 0.140 0.137 0.020 -0.227 0.365 

2 2,590 174 0.336 0.342 0.129 -0.131 0.277 
Credential completion: associate's degree 

1 3,914 641 0.109 0.110 0.012 -0.177 0.557 

2 2,336 158 0.292 0.303 0.094 -0.244 0.336 
Credential completion: certificate 

1 402 85 0.194 0.211 0.039 0.453 0.613 

2 254 16 0.453 0.342 0.281 -1.210 0.864 
Credits accumulated 

1 4,312 728 9.067 9.447 14.009 -0.399 0.406 

2 2,590 190 16.657 18.281 23.227 1.328 1.380 


Notes: Results are from a linear regression model (for continuous outcomes) orlogit model 
(for dichotomous outcomes) that accommodates for clustering of students within colleges. 


* Coefficient (of estimated intention-to-treat impact) is statistically significant at the 95 
percent confidence level using a two-tailed test. 
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their real-world implications. EDU operates the Regional Educational 
Laboratory Appalachia, funded by the U.S. Department of Education’s 
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